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[Name of Document] SPECIFICATION 422 Rec'd PCT/P7D 1 2 OCT 2000 

[Title of the Invention] 

Curve Generating Apparatus and Curve Generating Method, and Program Providing 
Medium 
[Claims] 
[Claim 1] 

A curve generating apparatus adapted for generating, from curve to be 
generated or processed of first frame, curve at second frame, 
the apparatus comprising: 

correspondence point detecting means for determiiiing, by using the curve to be 
generated of the first frame, correspondence point at the second frame corresponding 
to reference correspondence point set on the curve to be generated of the first frame; 
and 

curve generating means for generating curve of the second frame passing 
through the correspondence point detected by the correspondence point detecting 
means. 
[Claim 2 ] 

The curve generating apparatus as set forth in claim 1, wherein the 
correspondence point detecting means carries out picture image pursuit of reference 
correspondence point set on the curve to be generated of the first frame to determine 
correspondence point at the second frame. 
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[Claim 3] 

The curve generating apparatus as set forth in claim 1, wherein the curve 
generating means is adapted so that when the first frame is caused to be frame at the 
time of start and frame at the time of end is caused to be third frame, it determines 
interpolated curve by linear interpolation from curve to be generated of the first frame 
and curve of the third frame to deform this interpolated curve into curve passing 
through the correspondence point detected by the correspondence point detecting 
means. 
[Claim 4] 

The curve generating apparatus as set forth in claim 1, wherein the curve 
generating means generates, as curve, shape along edge 

of picture image passing through correspondence point detected by the 
correspondence point detecting means. 
[Claim 5] 

The curve generating apparatus as set foth in claim 1, wherein the curve 
generating means generates contour curve of object on picture image. 
[Claim 6] 

A curve generating method for generating, from curve to be generated or 
processed of first frame, curve at second frame, 
the method comprising: 

a correspondence point detection step of detenriining, by using curve to be 
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generated of the first frame, correspondence point at the second frame corresponding 
to reference correspondence point set on the curve to be generated of the first frame; 
and 

a curve generating step of generating curve of the second frame passing through 
the correspondence point detected by the correspondence point detection step. 
[Claim 7] 

The curve generating method as set forth in claim 6, wherein, at the 
correspondence point detection step, a procedure is taken to carry out picture image 
pursuit of reference correspondence point set on the curve to be generated of the first 
frame to detemiine correspondence point at the second frame. 
[Claim 8] 

The curve generating method as set forth in claim 6, wherein, at the curve 
generating step, a procedure is taken such that when the first frame is assumed to be 
frame at the time of start and frame at the time of end is assumed to be third frame, 
interpolated curve is determined by linear interpolation from the curve to be generated 
of the first frame and curve of the third frame to deform this interpolated curve into 
curve passing through the correspondence point detected at the correspondence 
detection step. 
[Claim 9] 

The curve generating method as set forth in claim 6, wherein, at the curve 
generating step, a procedure is taken to generate, as curve, shape along edge of picture 
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image passing through correspondence point detected at the correspondence point 
detection step. 
[Claim 10] 

The curve generating method as set forth in claim 6, wherein, at the curve 
generating step, a procedure is taken to generate contour curve of object on picture 
image. 
[Claim 11] 

A program providing medium adapted so that there is provided program 
relating to curve generating processing for generating curve at second frame from 
curve to be generated or processed of first frame, 

the program comprising: 

a correspondence point detection step of detennining, by using curve to be 
generated of the first frame, correspondence point at the second frame corresponding 
to reference correspondence point set on the curve to be generated of the first frame; 
and 

a curve generation step of generating curve of the second frame passing through 
the correspondence point detected by the correspondence detection step. 
[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 

This invention relates to a curve generating apparatus and a curve generating 
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method used in CAD, graphics, picture editing, etc. which edit curve indicating 
contour shape of object included within picture image on the computer, and a program 
providing medium adapted so that program relating to the curve generating method is 
provided. 
[0002] 
[Prior Art] 

Hitherto, there are many instances where contour extraction processing for 
extracting contour shape of object included in plural picture image data constituting 
moving picture image are used in the field of picture image processing, e.g., CAD, 
graphics technologies, etc. Among these picture image processing, when picture 
image synthesis processing is carried out, there is the processing which generates key 
signals from contour shape of the object, and it is important to generate precise or 
accurate key signals, i.e., precise contour shape. In this case, the key signal is 
information for cutting off the area of the foreground object to be synthesized and is 
also called mask. 
[0003] 

The contour extraction processing is required to have interactive operability as 
in the case where operator gives in detail information such as contour position or 
direction, etc. with respect to object. As such contour extraction processing, there are 
processing to designate plural points on the contour of object to prepare shapes 
between respective points, processing to operate control points of parametric curve 

5 



indicating counter shape, and processing to directly input contour shape by mouse, etc. 
As examples of these contour extraction processing, there are known "Intelligent 
Scissors for Image Composition" (Eric N. Mortensen and William A. Barrett, 
Computer Graphics Proceedings, Annual Conference Series, 1 995, ACM SIGGRAPH, 
pp. 191-198), "Image contour detection method" (Japanese Patent Application Laid 
Open No. 152481/1992 publication), and "Cutting mask preparation method and 
cutting mask preparation apparatus" (Japanese Patent Application Laid Open No. 
254854/1992 publication). 
[0004] 

Moreover, in the contour extraction processing, it is required to get precise 
contour shape with respect to all picture images constituting moving picture. For 
example, in cinemas or television picture images, several hundreds of key signals are 
required for generating images of several seconds, and the quantity of processing is 
vast. For this reason, the contour extraction processing used in the field of cinema, 
etc. is desired (required) that more precise contour shape can be prepared by lesser 
number of operations. This results from the fact that, in the conventional contour 
extraction processing, processing for preparing contour shapes with respect to 
respective frames constituting moving picture is the processing which requires 
extremely much time and labor. 
[0005] 

In view of the above, as a method of preparing contour shape by more simple 
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procedure, there was proposed a method in which, on the premise that, in moving 
picture images having continuous property, contour shape at the start frame changes 
with passage of time so that it changes to contour shape at the end frame, intermediate 
shape of contour at the intermediate frame is prepared by interpolation processing 
from contour shape at the start frame and contour shape at the end frame. 
[0006] 

In accordance with such an interpolation processing, as shown in FIG. 35, 
plural points constituting contour shape are caused to have one-to-one correspondence 
relationship at contour shape of the start frame and contour shape of the end frame. 
Namely, when it is assumed that curve indicating contour shape at the start frame is 
curve 0, curve indicating contour shape at the end frame is curve 1, the time at the 
start frame is 0, the time at the end frame is 1, and curve changes in point of time, it 
can be considered that curve 0 at time 0 changes to curve 1 at time 1. In view of the 
above, when time at which interpolation of two curves is carried out is assumed to be 
T, and coordinate at the point on the curve 0 is put by A and coordinate at the point on 
the curve 1 corresponding to the coordinate A is put by B, coordinate C at required 
intermediate form can be represented as follows. 

C= T • A + (1-T) • B 
In the conventional interpolation processing, contour at the intermediate frame was 
prepared from train of points determined by such interpolating calculation. 
[0007] 
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Further, it is assumed that contour shapes with respect to respective frames are 
constituted by plural third-order Bezier curve. The third-order Bezier curve is defined 
by 

Q(tH(l-t) 3 )M+3t((l-t) 2 )N+3(t 2 )(l-t)0+(t 3 )P (0<t<l) 
as indicated by the article "Computer Graphics PRINCIPLE AND PRACTICE 
SECOND EDITION in C" (Foley, van Dam, Feiner, Hughes, ADDISON WESLEY, 
1996, ISBN 0-201-84840-6). In this formula, M, N, O, P are coordinates indicating 
points on the two-dimensional plane wherein M and P are end points of line segment, 
and N, O are control points. An example of line segment of the third order Bezier 
curve represented by the above-mentioned formula is shown in FIG. 36. In FIG. 36, 
curve indicated by solid line is the third-order Bezier curve, wherein M, P indicated 
by points of black circle are end points, and points N, O indicated by white circle are 
control points. 
[0008] 

In accordance with the formula which defines the above-described Bezier curve, 
locus of the third-order Bezier curve, i.e., shape is represented by coordinates of end 
points and control points indicated by points M, N, O, P and change of time t in FIG. 
36. 

[0009] 

When constituting unit of the third-order Bezier curve shown in FIG. 36 is 
assumed to be segment, it can be said that single curve indicating contour shape is 
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ordinarily composed of plural segments as shown in FIG. 37. In this case, end points 
of adjacent segments are shared so that one curve is constituted as the entirety. 
[0010] 

[Problems to be Solved by the Invention] 

In the above-described conventional contour extraction processing, in the case 
where intermediate shape is detenriined from contour shapes of frames before and 
after in point of time, interpolation was carried out in the state where control points 
and end points constituting segments are caused to have one-to-one correspondence 
relationship. 
[0011] 

However, at frames before and after in point of time, there are instances where 
contour shape does not linearly move. 
[0012] 

In the above-mentioned method, in the case where contour shape move non- 
linearly, shape different from contour shape was prepared. 
[0013] 

In order to avoid such problems, it is conceivable to carry out work to divide 
interval of shape in point of time between the time of start and the tune of end into 
narrower ranges to carry out processing for a second time, or to manually modify the 
shape, etc. For this reason, there took place the problem that labor of user was 
required in preparing shape, and there took place fine unevenness of shape because 
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shape discontinuously changing was prepared in the time axis direction owing to 

manual modification. 

[0014] 

This invention has been made in view of actual circumstances as described 
above, and its object is to provide a curve generating apparatus and a curve generating 
method in which, in the processing for preparing intermediate shape of two curves, 
even in the case where contour shape does not linearly move, it is possible to precisely 
pursue correspondence points on the contour, thus to prepare contour shapes with 
higher precision. 
[0015] 

[Means to Solve the Problem] 

For this reason, to solve the problem described above, the curve generating 
apparatus according to this invention is directed to a curve generating apparatus 
adapted for generating, from curve to be generated of first frame, curve at second 
frame, the apparatus comprising: correspondence point detecting means for 
determining, by using curve to be generated of the first frame, correspondence points 
at the second frame corresponding to reference correspondence points set on the curve 
to be generated of the first frame; and curve generating means for generating curve of 
the second frame passing through correspondence points detected by the 
correspondence point detecting means. 
[0016] 
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Moreover, to solve the problem described above, the curve generating method 
according to this invention is directed to a curve generating method for generating, 
from curve to be generated of the first frame, curve at second frame, the method 
comprising: a correspondence point detection step of determining, by using the curve 
to be generated of the first frame, correspondence points at the second frame 
corresponding to reference correspondence points set on the curve to be generated of 
the first frame; and a curve generation step of generating curve of the second frame 
passing through the correspondence points detected by the correspondence detection 
step. 
[0017] 

In addition, to solve the problem described above, the program providing 
medium according to this invention is directed to a program providing medium for 
providing program relating to curve generation processing for generating, from curve 
to be generated of first frame, curve at second frame, wherein there is provided 
program comprising: a correspondence point detecting step of detennining, by using 
the curve to be generated of the first frame, correspondence points at the second frame 
corresponding to reference correspondence points set on the curve to be generated of 
the first frame; and a curve generation step of generating curve of the second frame 
passing through the correspondence points detected by the correspondence point 
detection step. 
[0018] 
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[Preferred Embodiment of the Invention] 

Preferred embodiments of this invention will now be described in detail with 
reference to the attached drawings. 
[0019] 

A curve generating apparatus 1 to which this invention is applied is caused to 
be of configuration as shown in FIG. 1, for example. This curve generating apparatus 
1 is adapted so that an arithmetic processing unit 2, a program memory 3, a data 
memory 4, a frame memory 5, an operation input unit 6, and an external storage 
(memory) unit 7 are connected to a bus 8. 
[0020] 

In the program memory 3, there is recorded program for executing a curve 
generating method according to this invention. Namely, the program memory 3 
records therein, as software program, the curve generating method comprising a 
correspondence point detection step of determining, by using curve to be generated or 
processed of the first frame, correspondence points at second frame corresponding to 
reference correspondence points set on the curve to be generated of the first frame, 
and a curve generation step of generating curve of the second frame passing through 
correspondence points detected by the correspondence point detection step. 
[0021] 

The curve generating apparatus 1 includes the operation input unit 6 which is 
comprised of, e.g., keyboard and/or mouse, etc. and serves to generate an operation 
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input signal as the result of the fact that it is operated by user. This operation input 
unit 6 outputs the operation input signal to the arithmetic processing unit 2. 
[0022] 

The arithmetic processing unit 2 generates control signals with respect to the 
above-described respective units in accordance with operation input signals from the 
operation input unit 6 . Further, the arithmetic processing unit 2 outputs control signals 
thus generated to respective units to thereby carry out processing corresponding to the 
operation input signals. Namely, the arithmetic processing unit 2 conducts such a 
control to read out in sequence the software programs from the program memory 3 to 
execute them to generate contour curve indicating contours of curves to be generated 
included in respective frames by using picture data indicating moving picture stored 
in the external storage (memory) unit 7. 
[0023] 

Picture image data processed by the software program from the program 
memory 3 is stored in the external storage (memory) unit 7. The external storage 
(memory) unit 7 comprises recording medium, e.g., optical disc, etc. and outputs, in 
accordance with control signal from the arithmetic processing unit 2, picture data 
indicating moving picture recorded on the recording medium to the arithmetic 
processing unit 2, the data memory 4 and the frame memory 5. In addition, the 
external storage (memory) unit 7 records the picture data onto the recording medium. 
[0024] 
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As shown in FIG. 2, for example, picture image data stored in the external 
storage (memory) unit 7 consists of plural frames and shows moving picture. In this 
example, the start frame is the Fr 1 , for example, and the end frame with respect to the 
frame Frl is frame Fr3. Namely, this curve generating apparatus 1 executes the 
program to thereby generate intermediate curve with respect to frame Fr2 which is 
intermediate frame by using first contour curve of data to be generated included in the 
frame Fr 1 and second contour curve of data to be generated included in the frame Fr3 . 
[0025] 

In the data memory 4, there is stored contour curve indicating contour of 
intermediate frame generated by the above-described arithmetic processing unit 2 
according to the program. In addition, into this data memory 4, data in the middle of 
arithmetic processing according to the program is also input. 
[0026] 

Picture image data corresponding to one frame from the data memory 4 or the 
external storage (memory) unit 7, etc. is input to the frame memory 5 in accordance 
with control signal of the arithmetic processing unit 2. Picture image data which has 
been read out by control signal from, e.g., arithmetic processing unit 2 or contour 
curve indicating intermediate frame generated by the program, etc. is inputted to the 
frame memory 5. Further, this frame memory 5 is connected to a display unit 9 for 
displaying picture image and/or processing content, etc. to user, and outputs picture 
image data, etc. inputted in accordance with control signal to the display unit 9 to 
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thereby allow the display unit 9 to display thereon picture image that the above- 
mentioned picture image data indicates. 
[0027] 

The program to which the curve generating method of this invention is applied 
is operated by the processing procedure shown in FIG. 3. 
[0028] 

At step S 10 1, user gives, as input, two curves at the time of start and at the time 
of end and reference correspondence points with respect to respective curves. In this 
case, two curves are respectively referred to as curves A and B. It is required that the 
numbers of reference correspondence points are the same with respect to the curves 
A and B. Then, at step SI 02, pursuit operation of reference correspondence points is 
carried out with respect to picture images successive in the time axis direction. As a 
result, positions of reference correspondence points at respective intermediate stages 
can be determined. The detail thereof will be described later. 
[0029] 

Then, at step SI 03, interpolation of shape is carried out. Interpolation 
calculation of shape is employed to carry out interpolation calculation of coordinates 
of point trains on two curves to thereby determine coordinates of points of 
intermediate shapes. Although there may be employed any transformation 
(conversion) to provide intermediate shape of two shapes, there is employed in this 
example the technology disclosed in the Japanese Patent Application No. 25 1 492/ 1 998 
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by the applicant of this application. The detail thereof will be described later. 
[0030] 

Then, at step S104, the interpolated shape determined at the step S103 is 
deformed in correspondence with reference correspondence points determined by 
pursuit operation at the step SI 02. 
[0031] 

Accordingly, by executing the program, it can be equivalently said that the 
arithmetic processing unit 2 comprises a correspondence point detecting section for 
determining, by using curve to be generated or processed of the first frame, 
correspondence points at intermediate frame corresponding to reference 
correspondence point set on the curve to be generated of frame at the time of start, and 
a curve generating section for generating curve of intermediate frame passing through 
correspondence point detected by the correspondence point detecting section. 
[0032] 

Particularly, the above-mentioned correspondence point detecting section 
carries out picture image pursuit operation at the step SI 02 to detect where reference 
correspondence point on curve to be generated or processed of frame at the time of 
start is positioned at the intermediate frame. 
[0033] 

In addition, the curve generating section generates curve passing through 
correspondence points detected by the correspondence point detecting section through 
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the steps S103 and S104. In more practical sense, when frame at the time of end is 
assumed to be the third frame, interpolated curve is determined by linear interpolation 
from curve to be generated or processed of the first frame and curve to be generated 
or processed of the third frame to deform the interpolated curve thus obtained into 
curve passing through correspondence points detected at the correspondence point 
detecting section. 
[0034] 

Operations at the respective steps of FIG. 3, i.e., processing that the arithmetic 
processing unit 2 carries out will now be described in detail. 
[0035] 

At step SI 02, there is carried out an operation to pursue position of reference 
correspondence points in picture images successive on the time axis. As a pursuit 
method of picture image, there may be employed any methods of pursuing points on 
the picture image. Here, explanation will be given with reference to FIG. 4 in 
connection with a method of carrying out pursuit by using absolute value sum of 
differences of picture images, which is known as SSD as an example. When picture 
image around reference correspondence point 10 at pursuit source frame F0 shown in 
FIG. 4 (A) is assumed as pursuit source image 20, pursuit destination images 40 are 
prepared with respect to all coordinates within the region of search range 30 
designated at frame Fl which is pursuing at present shown in FIG. 4 (B) to detennine 
by calculation analogousness between two images. As a method of determining 
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analogousness between images, there is used a method of deterniining absolute value 
sums of differences of respective pixel values within image region with respect to two 
images to allow them to be evaluation value to allow coordinates having the lowest 
evaluation value within the region to be pursuit result. 
[0036] 

Explanation will now be described in connection with interpolation calculation 
of shape at step SI 03 will be described below. In FIG. 5, there are shown two curves 
where the numbers of segments are not in correspondence with each other. An 
example for generating intermediate curve when change from first contour curve A to 
second contour curve B takes place using first contour curve A and second contour 
curve B will now be described. In this case, when end point, control point and line 
segment of third-order Bezier curve are assumed to be one segment, the first contour 
curve A consists of five segments and the second contour curve B consists of three 
segments. In addition, when intermediate curve in the case where change from first 
contour curve A to second contour curve B takes place as shown in FIG. 6, for 
example, is generated, procedure shown in the flowchart of FIG. 7 is taken. 
[0037] 

When the intermediate curve is generated, the operation input signal as the 
result of the fact that user operates keyboard, etc. is input designating correspondence 
points indicating corresponding positions of the first contour curve A and the second 
contour curve B. 

18 



[0038] 

Thus, at step SI, correspondence points are set at the first contour curve A and 
the second contour curve B in accordance with the operation input signal to the effect 
that the above-described correspondence points are designated to develop the first 
contour curve A and the second contour curve B into one straight line as shown in 
FIG. 8 to detenuine entire lengths of the first and second contour curves A and B 
which are closed curve to allow the respective entire lengths of the first and second 
contour curves A and B to be length A and length B, respectively. 
[0039] 

Then, at step S2, sampling intervals of the entire length A of the first contour 
curve A and the entire length B of the second contour curve B are detennined. 
Namely, such an approach is employed to compare magnitudes (sizes) of the length 
A and the length B shown at step S 1 1 of Fig. 9 and to locate the sampling points at the 
length interval which was resulted from dividing one longer in length by constant of 
the sampling interval. In this case, constant of the sampling interval is sampling 
interval where there can be obtained the number of points necessary for processing for 
re-constructing curve which will be described later at processing for generating such 
intermediate shape. Then, at step S12 of Fig. 9, at such an approach is employed to 
divide smaller one of magnitudes of length A and length B by the number of points 
within the section determined at the step SI 1 to obtain sampling interval. For this 
reason, sampling intervals less than the maximum sampling interval with respect to 
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both curves at all times can be obtained. 
[0040] 

Returning to FIG. 7, at step S3, resampling processing of curve A and curve B 
are carried out at sampling intervals determined at the step S2 to prepare point trains. 
Such state is shown in FIG. 10. Resampling processing are carried out with respect 
to the first contour curve A and the second contour curve B shown in FIG. 10 where 
sampling intervals are obtained. In this case, points on segment of one Bezier curve 
can be represented as a function of time t as described below. 

Q(t) = ((l-t) 3 )M+3t(l-t) 2 )N+3(t 2 )(l-t)0+(t 3 )P (0 < t < 1) 
In the above formula, M, N, O, P are coordinates indicating points on the two- 
dimensional plane, wherein M, P are end points of line segment, and N, O are control 
points. Further, in order to represent, by t, points on the Bezier curve constituted by 
connecting plural segments, definition of the Bezier curve is extended to allow the 
integral part of t to be segment No. and to allow the decimal part oft to be time within 
segment. Namely, in the case of t=3.45, this indicates coordinate on the curve of the 
third segment and at time of 0.45. Thus, points on the Bezier curve constituted by 
plural segments are represented by t. 
[0041] 

Moreover, detailed flowchart of resampling processing carried out at the step 
S3 of FIG. 7 is shown in FIG. 1 1 . At step S2 1 , when resampling processing is carried 
out, times at correspondence points of the first contour curve A and the second contour 
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curve B shown in FIG. 10 are set as Ta and Tb. At the subsequent step S22, 
coordinates of points on the first contour curve A and the second contour curve B at 
times Ta, Tb are determined. Respective points determined here result in sampling 
points at the first contour curve A and the second contour curve B. At the subsequent 
step S23, sampling interval determined at the above-described step S2 of FIG. 7 is 
added to times Ta, Tb. At the subsequent step S24, there is judged whether or not 
times Ta, Tb reach tenninating ends so that correspondence points return to 
correspondence points at the time of start of resampling processing. Further, when it 
is judged that correspondence point returns to correspondence point at the time of 
start, the processing is completed. On the other hand, when it is judged that 
correspondence point does not return to the correspondence point, the processing 
returns to step S22 to repeat processing of the above-described steps S22 to S24 to set, 
for a second time, sampling points at the first contour curve A and the second contour 
curve B to generate point train consisting of sampling points. Namely, in the 
resampling processing of the step S3 of FIG. 7, there are generated point train 
corresponding to the first contour curve A and point trains corresponding to the second 
contour curve B which consist of sampling points finer than those at the above- 
described sampling processing. 
[0042] 

Further, returning to FIG. 7, at step S4, such an approach is employed to carry 
out interpolation calculation of coordinates of point trains on two curves prepared at 
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the previously mentioned step S3 to thereby determine point train of intermediate 
shape. In more detail, processing shown in flowchart of FIG. 12 is carried out to 
thereby generate intermediate curve indicating intermediate shape when there takes 
place change from the contour shape that the first contour curve A indicates to contour 
shape that the second contour curve B indicates. In accordance with the flowchart of 
FIG. 12, first, at step S3 1, point train corresponding to the first contour curve A and 
point train corresponding to the second contour curve B which are determined by 
resampling processing of the step S3 of FIG. 7 are inputted from the data memory 4. 
Further, there are designated sampling points at positions corresponding to 
correspondence points on point train corresponding to the first contour curve A and 
point train corresponding to the second contour curve B from the correspondence 
points. At the subsequent step S32, when time at which interpolation of two curves 
is carried out is assumed to be T with respect to point train consisting of sampling 
points, coordinate of point on the first contour curve A is assumed to be a, and 
coordinate of point on the second contour curve B corresponding to the coordinate a 
is assumed to be b, coordinate C of the point at intermediate curve indicating the 
intermediate shape to be determined is expressed by the following formula (2). 

C = T-a + (l-T)-b (2) 
Namely, by applying the above-mentioned formula (2) to point trains corresponding 
to first and second contour curves, coordinate of sampling points constituting 
intermediate curve is determined from sampling points corresponding to the first 
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contour curve A and sampling points corresponding to the second contour curve B 
corresponding to sampling points corresponding to the first contour curve A. At the 
subsequent step S33, processing for designating next sampling points of the sampling 
points where the formula (2) is applied at the above-described step S32 is carried out 
with respect to point train corresponding the first contour curve A and point train 
corresponding to the second contour curve B. At the subsequent step S34, the 
arithmetic processing unit 2 judges whether or not correspondence point returns to the 
correspondence point where generation processing of intermediate curve is started at 
the above-described step S31 as the result of the fact that next sampling point is 
designated at the above-described step S33. Further, when it is judged that 
correspondence point returns to the corresponding point, the processing proceeds to 
step S3 5. On the other hand, when it is judged that correspondence points does not 
return to the correspondence point, the processing which has been explained at the step 
S32 is carried out with respect to sampling points designated at the step S33 . Namely, 
the processing which has been explained at the step S32 is carried out with respect to 
all of sampling points set at the above-described sampling processing of the step S3 
to thereby determine coordinate of points at the intermediate curve in accordance with 
the above-mentioned formula (2). At step S35, by using sampling points determined 
at the above-described step S3 and coordinate C of point at the intermediate curve 
determined at the step S32, the technology shown in the Japanese Patent Application 
Laid Open No. 164436/1998 publication or "An Algorithm for Automatically Fitting 
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Digitized Curves (pp. 612-626) in the "Graphics Gems" (Andrew S. Glassner) is 
applied to thereby transform (convert) point train consisting of sampling points into 
Bezier curve. Thus, curve is reconstructed from point train consisting of sampling 
points corresponding to first contour curve A and point train consisting of sampling 
points corresponding to the second contour curve B to generate intermediate curve 
indicating intermediate shape when change from first contour curve A to second 
contour curve B takes place as shown in FIG. 6. 
[0043] 

The curve generating apparatus 1 for carrying out such an intermediate curve 
generation processing is adapted so that when, as shown in FIG. 5, intermediate curve 
when there takes place change from the first contour curve A to the second contour 
curve B is generated, even if the numbers of segments of the first contour curve A and 
the second contour curve B are different from each other, the first and second contour 
curves are caused to undergo sampling to set the same number of sampling points with 
respect to the first contour curve A and the second contour curve B to re-construct 
curve. Accordingly, it is possible to generate intermediate curve as shown in FIG. 6 
with high quality and simply. 
[0044] 

Here, returning to FIG. 3, at step S104, interpolated shape determined at step 
S 1 03 is deformed in correspondence with reference correspondence points determined 
at the step S 102. Explanation will be given with reference to FIGS. 13, 14 and 15 with 
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respect to such an operation. 
[0045] 

First, at step S41 of FIG. 13, start point 115 and end point 1 16 are caused to 
correspond to each other with respect to reference correspondence points 110 
indicated at START of FIG. 14 determined by image pursuit processing of the step 
S102 of FIG. 3 and reference correspondence points 100 indicated at START of FIG. 
14 determined at the intermediate shape preparation processing of the step SI 03. It 
is now assumed that coordinates of start points at the step S103 are A (exl, eyl) and 
B(ex2, ey2) and coordinates of end points at the step S102 are C(sxl, syl) and D(sx2, 
sy2). At the step S42, such a coordinate transformation (conversion) to convert line 
segment AB into line segment CD is determined. Such a transformation can be 
represented by combination of parallel displacement and rotation, and operation of 
scaling. Such a transformation is called affine transformation 120. 
[0046] 

The transform formula of affine transformation is as follows. 

[0047] 

With respect to the method of transforming point (x, y) into point (x\ y') by 
the affine transformation, the following relational expression is assumed to hold, 
[x'] = sc* [cosO - sin9][x-exl] + [sxl] 
[y'] = sc* [sinO - cos0][y-eyl] + [syl] 

[0048] 
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In the above formula, sc = L1/L2 
LI = V((sx2-sxl) A 2 + (sy2-syl) A 2) 
L2 = V((ex2-exl) A 2 + (ey2-eyl) A 2) 
6=02-01 

61 = a tan 2 ((sy2-syl)/Ll, (sx2-sxl)/Ll) 

62 = a tan 2 ((ey2-eyl)/Ll, (ex2-exl)/L2) 

[0049] 

Then, at the step S43, affine transformation determined at the step S42 is 
calculated with respect to point train constituting intermediate shape determined at the 
step S102 of FIG. 3. As a result, there is prepared shape which holds feature of 
intermediate shape determined at the step S103 and passing through end point 
determined at the step 102. Then, at step S44, by using the method of transforrriing 
(converting) point train detennined at step S43 into the previously described Bezier 
curve, transformation into curve is carried out. By carrying out such a processing with 
respect to all sections of shape, it becomes possible to transform shape prepared at the 
intermediate shape preparation processing of the step S103 into shape passing through 
reference correspondence points prepared at the step S102. 
[0050] 

As described above, in accordance with this invention, in preparing intermediate 
shape of two curves, it becomes possible to prepare intermediate shape passing 
through designated reference correspondence points. In addition, it is possible to set 
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reference correspondence point at precise position by image pursuit processing. As 
a result, it has become possible to prepare, with high precision, intermediate shape 
determined at picture image designated on the time axis. Thus, positional shift 
between shape by linear interpolation and actually determined shape which was the 
problem of the prior art can be solved. 
[0051] 

Another more practical example of this invention will now be described. In 
FIG. 16, there is shown the example in which the step S103 is omitted among the 
condition of the first embodiment shown in FIG. 3 and the portion of the step SI 04 is 
realized by another method as step S105. 
[0052] 

Edge information of picture image is used as substitutive means for preparing 
intermediate shape to carry out processing for automatically generating shape passing 
through the reference correspondence point and along the edge. In more practical 
sense, as a method of detecting optimum path passing through the designated point, 
there is used the technology of path search method disclosed in the Japanese Patent 
Application No. 204862/1997 by the applicant of this application. 
[0053] 

Explanation will be first given in connection with curve generating means which 
generates curve by the path search method disclosed in the Japanese Patent 
Application No. 204862/1997. 
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[0054] 

FIG. 17 is a block diagram showing curve generating means. The curve 
generating means is composed of respective processing means composed of two point 
selector means 151, path search processing optimization means 152, path search 
means 153 and curve approximate means 154, and respective data stores (storages) 16, 
17, 23, 18 for point train, picture image, path and curve. 
[0055] 

The two point selector means 151 selects successive two points from the point 
train data storage means 16 to output them. The path search processing optimization 
means 152 calculates path search parameter for optimizing the path search method 
from the selected two points and picture image data from the picture data storage 
means 17 to send it to the path search means 153. 
[0056] 

The path search means 153 determines path search processing method on the 
basis of the path search parameter that the path search processing optimization means 
1 52 has calculated to calculate path connecting the selected two points and passing on 
the contour within picture image to hold it by the path data storage means 23. 
[0057] 

In this case, path data is list of pixel coordinates indicating eight vicinity 
graphics connecting the above-mentioned two points. As indicated by broken lines in 
FIG. 1 7, the path search means 153 sends control information to the two point selector 
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means 15 1 to select the next two points. Accordingly, path searches are carried out 
with respect to portions between respective points of the point train storage means 16 
so that all the search data are accumulated (stored) into the path data storage means 
23. 

[0058] 

When the two point selector means 15 1 has selected until the last of the point 
train to fail to output new two points, it sends control information to the curve 
approximate means 1 54 as indicated by broken lines in FIG. 17 to hasten the operation 
of the curve approximate means 154. The curve approximate means 154 carries out 
curve approximation of path stored in the path data storage means 23 to prepare curve 
data indicating contour shape to hold it by the curve data storage means 18. 
[0059] 

Explanation will now be given with reference to FIGS. 18 and 17 in connection 
with path search method that the curve generating means executes. 
[0060] 

FIG. 18 is a flowchart showing fundamental processing procedure of the path 
search method according to this invention. First, at step SI, the two point selector 
means 151 selects successive two points from the point train data storage means 16 to 
output them. 
[0061] 

Then, at step S2, optimization of path search parameter is carried out. Path 
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search parameter is calculated by the path search processing optimization means 152 
on the basis of the two points selected by the point selector means 151 and picture 
data. 
[0062] 

Then, at step S3, path search is carried out. The path search means 153 
determines the path search method on the basis of path search parameter obtained at 
the second step S2 to extract contour connecting the above-mentioned two points in 
the form of 8 vicinity path. 
[0063] 

Further, at step S4, there is carried out judgment as to whether or not path 
searches between all relaying (repeating) points have been completed. As a result, if 
such path searches have not yet been completed, the processing returns to step SI to 
select two points in which one point is shifted from the two points selected at last time 
to repeat steps S2, S3. On the other hand, when it is judged at step S4 that path 
searches between all relaying points have been completed, the processing proceeds to 
step S5. 
[0064] 

At the step S5, contour curve which approximates 8 vicinity path is generated 
to complete the processing. The curve approximate means 154 carries out curve 
approximation of path data stored in the path data storage means 23 to prepare curve 
data indicating contour shape. 
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[0065] 

In this case, the curve approximate means 154 can be realized by the technology 
in which the applicant of this application has disclosed "Curve Generating Apparatus 
and Curve Generating Method" in the Japanese Patent Application Laid Open No. 
303687/1996 publication. This technology is a method of obtaining curve which 
approximates coordinate list and is one of the so-called curve approximation methods. 
Namely, this method is method of approximating order of input data while dividing it 
within the sufficiently approximatable range so that this method is robust to noise and 
has good efficiency as compared to the conventional curve approximations. 
[0066] 

However, the above-mentioned curve approximation means 154 may be 
methods of curve-approximating a series of coordinate lists, and may be realized by 
a method of generating B-spline curve generally used, etc. 
[0067] 

It is to be noted that general curve approximation/interpolation methods are 
described in detail in articles such as "Numerical Calculation Handbook" (Yutaka 
Ohno, Kazuo Isoda, Ohm Company, 1990), "Shape Processing Engineering [I] [II] by 
Computer Display" (Fujio Yamaguchi, NikkanKogyo Shinbun Sha, 1982, "Computer 
Graphics Principles and Practice" (Foly, vanDam, Feiner, and Hughes, Addison 
Wesley Company, 1990), etc. 
[0068] 
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The path search processing optimization means 1 52 and the path search means 
153 will now be described in more detail. 
[0069] 

FIG. 19 is a block diagram in which the portion for carrying out path search is 
extracted of the curve generating means. The feature of the curve generating means 
for carrying out this path search resides in that path search processing optimization 
means 152 is provided to thereby carry out path search processing optimized with 
respect to inputted two points. In this example, the path search processing 
optimization means 152 is adapted to output two kinds of path search parameters 
which will be described later. 
[0070] 

FIG. 20 shows the internal configuration of the path search processing 
optimization means 152. The path search processing optimization means 152 is 
composed of passing cost calculation range detenriining means 156 and passing cost 
calculation optimization means 15 7. The passing cost calculation range detennining 
means 156 inputs selected two points to output calculation range parameter. Further, 
the passing cost calculation optimization means 157 outputs passing cost calculation 
parameter from the selected two points and picture data. In addition, these two kinds 
of path search parameters are inputted to the path search means 153 and are used for 
determining the path search method. 
[0071] 
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FIG. 21 is a flowchart showing processing procedure at the path search 
processing optimization means 152. 
[0072] 

First, at step SI 1, optimization of passing cost calculation parameter is carried 
out. In the passing cost calculation optimization means 157 calculates passing cost 
calculation parameter from the selected two points (point 1 and point 2) and picture 
image. Then, at step S 1 2, optimization of the passing cost calculation range parameter 
is carried out to complete the processing. The passing cost calculation range 
deteranning means 156 inputs selected two points to calculate calculation range 
parameter. 
[0073] 

FIG. 22 is a block diagram showing, in more detail, the portion for carrying out 
processing relating to passing cost calculation parameter in FIG. 20. 
[0074] 

The passing cost calculation optimization means 157 is caused to be of 
configuration comprising gradient parameter deteraiining means 158. 
[0075] 

Moreover, the path search means 153 is caused to be of configuration 
comprising passing cost calculation means 159, niinimum cost path calculating means 
161, and passing cost map storage means 1 60 for storing data called pas sage cost map 
which will be described later. 
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[0076] 

The gradient parameter detennining means 158 inputs picture data and two 
points to calculate gradient parameter optimum for extracting contour between the two 
points to output it to the passing cost calculating means 159 of the path search means 
153. 
[0077] 

The passing cost calculating means 159 inputs picture data, the point 1, the 
gradient parameter, and calculation range parameter that the passing cost calculation 
range detennining means 1 56 shown in FIG. 20 outputs to calculate passing cost map. 
[0078] 

The minimum cost path calculating means 161 inputs passing cost map and the 
point 2 to calculate path connecting the points land 2 so as to trace the contour 
therebetween. 
[0079] 

FIG. 23 is a flowchart showing processing procedure at the path search means 

153. 
[0080] 

First, at step S21, the passing cost calculating means 159 described later 
calculates passing cost map. Then, at step S22, the minimum cost path calculating 
means 161 calculates the minimum cost path. The detail of this operation will be 
described later. 
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[0081] 

FIG. 24 shows, in more detail, the configuration of the portion for carrying out 
processing relating to the above-described passing cost calculation parameter. 
[0082] 

The passing cost calculation parameter is two kinds of parameters necessary m 
calculating gradient of picture image. The first parameter is normal vector indicating 
direction perpendicular to advancing direction of contour, and the second parameter 
is pixel value color change vector indicating change of pixel value when observed in 
a direction traversing the contour. 
[0083] 

Moreover, the path search means 153 carries out a processing to extract, by 
utilizing the fact that object contour has high gradient strength, such a path on the 
picture image to pass as close as possible the portion having higher gradient strength. 
[0084] 

In order to carry out processing as described above, the pass cost calculating 
means 159 is composed of gradient calculating means 164, passing cost map 
calculating means 165 and gradient data storage means 21. 
[0085] 

The gradient calculating means 164 calculates gradient of picture image, and 
the passing cost map calculating means 165 calculates passing cost map such that 
when path is passed through the portion where gradient strength is higher, cost is 
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reduced to more degree on the basis of the gradient thus obtained. Position of the 
point lis inputted to the passing cost map calculating means 165 and starting point of 
the passing cost map is set at the point 1. 
[0086] 

In this example, the gradient calculating means 164 selectively detects only 
gradient component of necessary contour by using the technology in which the 
applicant of this application has disclosed in "Edge Detection Method and Edge 
Detection Apparatus" in the Japanese Patent Application Laid Open No. 1 34420/ 1995 
publication. By this gradient calculating means 164, gradient is calculated so as to 
selectively extract gradient of contour extending in a manner connecting inputted two 
points. 
[0087] 

Explanation will now be given in detail in connection with calculation process 
of the above-described two kinds of path search parameters. 
[0088] 

First, calculation of passing cost calculation parameter and the process where 
such parameter is utilized will be described. FIG. 25 is a flowchart showing 
processing procedure at the gradient parameter detenriining means 158. At step S3 1 , 
normal vector calculating means 162 calculates normal vector perpendicular to the 
direction connecting inputted points 1 and 2 to output it. At step S32, pixel value 
change vector calculating means calculates change of pixel value of picture image 
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when observed in a direction perpendicular to the direction connecting two points, e.g., 
color change direction in the neighboring region of the points land 2 on the picture 
image to output it as color change vector. 
[0089] 

In this example, the passing cost map is information in which, in each pixel, 
direction admitted to that pixel position at minimum passing cost is recorded. 
[0090] 

FIG. 26 shows an example of simple passing cost map. 

[0091] 

The passing cost map is prepared with respect to one range within picture image 
designated by the passing cost calculation range parameter. In this example, there is 
shown an example where such map is prepared in a certain small quadrilateral region. 
[0092] 

In FIG. 26(A), directions admitted to that pixel position at the minimum passing 
cost in the respective pixels are indicated by arrow. This direction is calculated so that 
only one direction is determined in each pixel. Value labeled at each pixel is 
accumulated value of passing cost when admitted in accordance with the minimum 
passing cost direction. This accumulated value is determined by allowing point 1 
serving as starting point of path search as accumulated value 0. 
[0093] 

FIG. 26(B) shows process where minimum path calculating means 161 
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determines path when passing cost map is obtained as shown in FIG. 26(A). 
[0094] 

The minimum cost calculating means 161 inputs passing cost map and the point 
2 to trace, in an opposite direction, minimum passing cost direction of the passing cost 
map from the point 2 to thereby determine path of the minimum cost ranging from the 
point 1 to the point 2. For example, in the case where the point 2 is located at position 
shown in FIG. 26(B) (accumulated cost is 22), the arrows are traced from the point 
2 (written in a direction opposite to that in FIG. 26(A)) to determine path until the 
point 1 as indicated by slanting lines. 
[0095] 

Explanation will now be given in connection with means for calculating passing 
cost map as described above at a high speed. Calculation of passing cost map is 
generally known as a sort of optimization problem, and a method of calculating such 
a cost map by using dynamic programming, etc. is well known. Also in the previously 
described article "Intelligent Scissors for Image Composition," a method using 
dynamic prograrmiing is described. 
[0096] 

FIG. 27 shows an example of description relating to shortest path search 
algorithm using dynamic programming. It is to be noted that, in this description, signs, 
etc.. are partially changed from the program described in the above-mentioned article 
"Intelligent Scissors for Image Composition". 
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[0097] 

Calculation of calculation range parameter and process where such parameter 
is utilized will now be described. 
[0098] 

FIG. 28 is a block diagram showing, in more detail, the portion for carrying out 
processing relating to calculation range parameter in FIG. 20. 
[0099] 

The passing cost calculation range detenriining means 156 is caused to be of 
configuration comprising distance calculating means 167, calculation range width 
calculating means 168, calculation range length calculating means 169, and calculation 
range calculating means 170. 
[0100] 

The distance calculating means 167 inputs points 1 and 2 to calculate distance 
between the two points and direction connecting the two points. The calculation range 
length calculating means 169 inputs the distance between the two points obtained to 
calculate calculation range length. The calculation range width calculating means 168 
inputs distance between the two points obtained to calculate calculation range width. 
The calculation range calculating means 170 calculates calculation range parameter 
to determine shape of the calculation range on the basis of the above-mentioned 
calculation range length, calculation range width and direction connecting two points 
that the distance calculating means 167 has outputted to input it to the passing cost 
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calculation means 159 of the path search means 153. The passing cost calculation 
means 159 calculates passing cost map within the range indicated by the calculation 
range parameter. 
[0101] 

FIG. 29 is a flowchart showing processing procedure at the passing cost 
calculation range deteirnining means 156 of FIG. 28. 
[0102] 

First, at step S41, the distance calculating means 167 inputs points 1 and 2 to 
calculate distance between these two points and direction connecting two points. 
[0103] 

Then, at step S42, the distance between the two points is inputted. Thus, 
calculation range width is calculated. In this case, the calculation range width 
calculating means 168 has therein rule of magnitude (size) of calculation range width 
with respect to magnitude (size) of distance between two points in the form such as 
calculation formula or reference table, etc. to determine calculation range width in 
accordance with that rule. For example, the problem in which even if operator 
attempts to set relaying (repeating) points in detail at short intervals in order to obtain 
a desired contour, searched path is meandered between relaying points so that shape 
differs from desired shape of contour can be solved by employing a rule such that the 
rule for determining calculation range width changes to a rule such that according as 
distance between two points becomes smaller, width becomes smaller. 
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[0104] 

At this time, calculation range width between relaying (repeating) points set in 
detail at short intervals is set so that it becomes very narrow, and path search is carried 
out so that relaying (repeating) points are connected by substantially straight line. 
Thus, contour shape that operator desires is obtained. 
[0105] 

Then, at step S43, the calculation range length calculating means 169 calculates 
calculation range length from distance between two points. In this case, the 
calculation range length is determined so that it is larger than the distance between two 
points by margin length detenriined in advance. 
[0106] 

Finally, at step S44, the calculation range calculating means 170 calculates 
calculation range parameter from the calculation range width, the calculation range 
length and direction connecting two points. In this case, the calculation range is such 
a rectangle to surround inputted two points and inclined in the direction connecting 
two points, and the calculation range parameter indicates positions of four corners of 
that rectangle. 
[0107] 

The point train generating means 12 will now be described. 

[0108] 

The point train generating means 12 inputs curve to output point train 
41 



reproducing that shape and reconstructed curve reconstructed by that point train. The 
point train and the reconstructed curve which have been generated are used for curve 
editing work which will be carried out later by the configuration of the first 
embodiment. 
[0109] 

FIG. 30 is a block diagram showing an example of the configuration of the point 
train generating means 12. 
[0110] 

This point train generating means 12 is caused to be of configuration 
comprising initial point train generating means 171, curve reconstructing means 172, 
difference detecting means 173, point train editing means 14, picture image data 
storage means 17, point train data storage means 16, and reconstructed curve data 
storage means 19. 
[0111] 

The initial point train generating means 171 inputs input curve and picture 
image to generate initial point train. The point train is stored in the point train data 
storage means 16. The curve reconstructing means 172 inputs point train and picture 
image to generate such a curve to pass the point train and to pass on the contour of the 
picture image. 
[0112] 

The difference detecting means 1 73 inputs input curve and re-constructed curve 
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to output difference between both shapes as difference. The point train editing means 
14 operates point train data in the point train data storage means 16 on the basis of 
difference that the difference detecting means 173 has outputted. 
[0113] 

FIG. 31 is a flowchart showing processing procedure in the point train 
generating means 12. 
[0114] 

First, at step S51, the initial point train generating means 171 carries out 
extraction of feature points of input curve to store the extracted feature points into the 
point train data storage means 16 as initial point train. 
[0115] 

For the above-mentioned feature point extraction on the curve, there may be 
used feature point extraction method shown in "Motion Vector Estimation Method" 
described in the Japanese Patent Application Laid Open No. 335270/1996 publication. 
In accordance with this method, such an approach is employed to calculate very small 
section bending angles at respective points from the initial end to the terminating end 
of curve to allow its sum total to be accumulated bending angle, whereby when the 
accumulated bending angle is greater than a threshold value determined in advance, 
the point which takes the maximum very small section bending angle is caused to be 
feature point. Further, similar feature point extraction processing is recursively carried 
out with respect to curve divided by feature points. 
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[0116] 

Then, at step S52, the curve re-constructing means 172 calculates re-constructed 
curve by processing similar to the processing in the previously described path search 
apparatus according to this invention from the point train. 
[0117] 

Then, at step S53, the difference detecting means 173 calculates difference 
between input curve and re-constructed curve. In accordance with such difference 
detection processing, such an approach is employed to take difference detection points 
at suitable intervals on the re-constructed curve to measure distance up to input curve 
from that difference detection point to allow it to be difference value. 
[0118] 

Then, at step S54, the difference detecting means 173 examines whether or not 
there exist difference value greater than threshold value set in advance. If there does 
not exist difference value greater than the threshold value, the processing of the point 
train generating means 12 is completed. Thus, data in the point train data storage 
means 16 and re-constructed curve data storage means 19 at that time point serve as 
output. On the other hand, if there exists difference value greater than the threshold 
value, the processing proceeds to step S55. 
[0119] 

At the step S55, the point train editing means 14 edits point train data on the 
basis of difference thus obtained. At respective difference detection points in which 
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it is judged that the difference value is greater than threshold value, new relaying 
(repeating) point is placed at point on the input curve nearest from that difference 
detection point to store point train data into the point train data storage means 16. 
[0120] 

After completion of the step S55, the processing returns to the step S52. Thus, 
at step S54, processing is repeated until difference between input curve and re- 
constructed curve becomes smaller than threshold value. 
[0121] 

While, in the above-mentioned processing, the number of points constituting 
initial point train is set to lesser value at first to suitably supplement initial points at the 
shortage portions, there may be employed a method in which the number of points 
constituting initial point train is set to larger value to delete unnecessary points. In that 
case, such an approach may be employed to generate initial point train to such a degree 
that input curve can be sufficiently polygon-approximated by the initial point train 
generating means 171 to select and delete relaying (repeating) points in which 
difference value therearound is sufficiently smaller than threshold value by the 
difference detecting means 173 after re-construction of the curve. 
[0122] 

It is to be noted that while, in the processing procedure shown in FIG. 3 1 , since 
re-construction can be completed by lesser number of points, there results the merit 
that curve editing thereafter is easy, processing procedure which will be described later 
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has the merit that because relaying (repeating) point interval for carrying out path 

search can be small, processing time can be shortened. 

[0123] 

Moreover, point train obtained by the above-mentioned method is transformed 
(converted) into curve by using curve generating means. Namely, by using edge 
information of picture image in preparing intermediate shape passing through portions 
between respective correspondence points by using the above-mentioned method, it 
has become possible to prepare contour having high precision along the shape 
represented in the picture image. 
[0124] 

Further, in the method shown in FIG. 16, such an approach may be employed 
to determine edge of picture image by using another algorithm at step S105 to 
detenriine intermediate shape along the edge of that picture image. 
[0125] 

Namely, at step S102, a set of correspondence points determined by picture 
image pursuit and the picture image are used to determine color projection axis which 
maximizes contrast around the edge to form edge of picture image by using such color 
projection axis to determine intermediate shape along this edge. 
[0126] 

The configuration of optimum color projection axis calculating unit which 
inputs the set of correspondence points as representative points 1 and 2 to determine 
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the color projection axis is shown in FIG. 32. 
[0127] 

The optimum color projection axis calculating unit 32 is composed of a 
representative point vicinity (neighboring) color projection axis calculating section 
41 and a representative point vicinity (neighboring) optimum color projection axis 
calculating section 4 1 for calculating vicinity (neighboring) optimum color projection 
axes within the region in the vicinity of representative points 1 and 2, and a color 
projection axis synthesis section 42 for synthesizing color projection axis by using the 
color projection axis land the color projection axis 2 from the above-mentioned two 
calculating sections 4 land 41. 
[0128] 

At the representative point vicinity (neighboring) optimum color proj ection axis 
calculating sections 41, there are determined vicinity (neighboring) optimum color 
proj ection axes which maximize contrast around the edge included within neighboring 
regions of representative points. It is now assumed that, as shown in FIG. 33, 
representative points 1 (pO) and 2 (p 1 ) are determined. At this time, the representative 
point vicinity (neighboring) color projection axis calculating section 4 1 detennines, by 
calculation, color projection axis (1) BO in which the contrast is maximum of edge 
included within the neighboring region Al of representative point p 1 . In addition, the 
representative point vicinity neighboring color projection axis calculating section 41 
determines, by calculation, color projection axis (2) Bl in which the contrast is 
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maximum of edge included within the vicinity (neighboring) region A2 of the 

representative point pi. 

[0129] 

The color projection axis synthesis section 42 synthesizes vicinity (neighboring) 
optimum color projection axis (1) BO, vicinity (neighboring) optimum color projection 
axis (2) Bl respectively independently detennined within the vicinity (neighboring) 
regions of representative points land 2 to determine one color projection axis where 
the contrast is maximum of edge of updating section. As the synthesis method in this 
case, e.g., average vector of color projection axes may be axis after synthesis. 
[0130] 

In addition, the maximum color projection axis calculating section 32 may 
determine optimum color projection axis by the method shown in FIG. 34. 
[0131] 

As shown in FIG. 34(A), it is assumed that anchor point p0 and adjacent anchor 
point pi are placed on edge of picture image. At this time, since there is the possibility 
that the edge may deviate to much degree from the line segment connecting pO and p 1 , 
there is carried out search of region where color change is the greatest within the range 
of length H on the line segment in a direction perpendicular to pOp 1 with respect to 
points qO, ql, and q2 placed every equal interval with respect to line segment length 
L on the line segment pOpl, e.g., interval of L/4. 
[0132] 
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When color with respect to a certain point r is assumed to be C(r), it is defined 
that color change vector is dC/dr and its magnitude is jdC/dr|. Thus, r in which this 
value takes the maximum value is determined. 
[0133] 

As the result of search, rO, r 1 , r2 are assumed to be detennined as shown in FIG. 
34(B). At this time, color change vectors dC/dr are respectively determined at rO, rl, 
r2 to allow its average value to be optimum color projection axis. 
[0134] 

In addition, color change vector may be determined in a manner described 
below. In the case of rO, colors C(s00), C(s01) are detennined with respect to points 
sOO, sOl having rO as center and away from each other by distance width h on the 
straight line perpendicular to the line segment pOp 1 . Thus, color change vector is 
substituted for its difference vector aC=C(s01)-C(s00). In connection with rl, r2, 
color change vectors are respectively similarly determined. 
[0135] 

Ordinary contour closed curve generating unit generates contour closed curve 
on the basis of the optimum color projection axis which has been explained. At the 
step SI 05, intermediate shape based on this contour closed curve is prepared. 
[0136] 

In addition, when algorithm shown in FIG. 16 is deformed and linear 
interpolation at the step S 1 02 fails to be carried out because, e.g., movement of picture 
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image is suddenly carried out, the processing of the step SI 05 may be carried out. 
[0137] 

As described above, in accordance with this invention, positional shift between 
shape by linear interpolation and shape actually deteraiined, which was problem of the 
prior art, can be solved. 
[0138] 

[Effect of the Invention] 

As previously mentioned, in accordance with this invention, even in the case 
where contour shape does not linearly move, it is possible to precisely pursue 
correspondence points on the contour. Thus, contour shape having higher precision 
can be prepared. 

[Brief Description of the Drawings] 

FIG. 1 is a block diagram showing configuration of a curve generating apparatus 
to which this invention is applied. 

FIG. 2 is a view for explaining moving picture consisting of plural frames to be 
processed at the above-mentioned curve generating apparatus. 

FIG. 3 is a flowchart showing processing procedure in software program 
relating to a curve generating method of this invention. 

FIG. 4 is a view for explaining a method of carrying out pursuit by using 
absolute value sum of differences of picture images as pursuing method for picture 
image. 
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FIG. 5 is a view for explaining first contour curve A and second contour curve 
B used when intermediate curve is generated by the curve generating apparatus. 

FIG. 6 is a view showing intermediate curve generated by the curve generating 
apparatus when there takes place change from first contour curve A to second contour 
curve B. 

FIG. 7 is a flowchart showing processing of arithmetic processing unit when 
intermediate curve is generated and there takes place change from the first contour 
curve A to the second contour curve B. 

FIG. 8 is a view for explaining that sampling points are set to determine 
sampling interval. 

FIG. 9 is a flowchart showing processing when sampling intervals with respect 
to first contour curve A and second contour curve B are detennined. 

FIG. 10 is a view showing first contour curve A and second contour curve B 
from which sampling interval is obtained. 

FIG. 11 is a flowchart showing processing when resampling processing is 
carried out. 

FIG. 12 is a flowchart showing processing when there takes place change from 
first contour curve A to second contour curve B, sampling is required, and 
intermediate curve indicating intermediate shape is generated. 

FIG. 13 is a flowchart for explaining processing procedure for deforming 
interpolated shape in correspondence with reference correspondence points. 
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FIG. 14 is a view showing a more practical example of processing up to step 

542 of the flowchart shown in FIG. 13. 

FIG. 15 is a view showing a more practical example of processing at the step 

543 and steps subsequent thereto of the flowchart shown in FIG. 13. 

FIG. 16 is a flowchart for explaining operation of another more practical 
example of this invention. 

FIG. 17 is a block diagram showing an example of the configuration of the main 
part of curve generating means for generating curve by the more practical example of 
path search method which was disclosed in the Japanese Patent Application No. 
204862/1997. 

FIG. 18 is a flowchart showing basic processing procedure of the path search 
method. 

FIG. 19 is a view for explaining the part for carrying out path search on the 
basis of the path search method. 

FIG. 20 is a view for explaining internal configuration of path search processing 
optimization means. 

FIG. 2 1 is a flowchart for explaining processing procedure in the path search 
processing opthnization means. 

FIG. 22 is a view for explaining the part for processing relating to passing cost 
calculation parameter of the path search processing optimization means. 

FIG. 23 is a flowchart for explaining processing procedure in the path search 
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means. 

FIG. 24 is a view for explaining, in detail, configuration of the part for 
processing relating to passing cost calculation parameter utilizing selective gradient 
detection. 

FIG. 25 is a flowchart for explaining processing procedure in passing cost 
optimization means. 

FIG. 26 is a view showing one example of passing cost map of path search. 

FIG. 27 is a view showing one example of the shortest path search algorithm 
using dynamic prograniming. 

FIG. 28 is a view for explaining in detail the part for processing relating to 
calculation range optimization of path search processing optimization means. 

FIG. 29 is a flowchart for explaining processing procedure in calculation range 
optimization means. 

FIG. 30 is a block diagram showing an example of the configuration of point 
train generating means. 

FIG. 3 1 is a flowchart for explaining processing procedure in the point train 
generating means. 

FIG. 32 is a block diagram showing the configuration of a more practical 
example of optimum color projection axis calculating unit for determining color 
projection axis. 

FIG. 33 is a view for explaining the operation principle of a more practical 
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example of the optimum color projection axis calculating unit. 

FIG. 34 is a view for explaining the operation principle of another more 
practical example of the optimum color projection axis calculating unit. 

FIG. 35 is a view for explaining generating intermediate shape by interpolation 
processing from start frame and end frame in conventional technique of generating 
intermediate shape . 

FIG. 36 is a view showing an example of line segment of third-order Beizer 

curve. 

FIG. 37 is a view showing single curve indicating contour shape as plural 
segments. 

[Explanation of Referenced Numerals] 

1 curve generating apparatus; 2 arithmetic processing unit 
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The Live-Wire 2-D dynamic programming (DP) graph search algorithm is as follows: 
Algorithm: Live-Wire 2-D DP graph search. 
Input: 

s {Start(or seed) pixel.) 

I(q,r) {Local cost function for link between pixels q and r.} 
Data Structures: 

L {List of active pixeis sorted by total cost (initially empty).) 

N(q) {Neighborhood set of q (contains 8 neighbors of pixel).} 

e(q) {Boolean function indicating if q has been expanded/processed.} 

g(q) {Total cost function from seed point to q.} 

Output: 

p {Pointers from each pixel indicating the minimum cost path.} 

Algorithm: 

g(s)=0; L=s; {Initialize active list with zero cost seed pixel.} 

while L!=NULL do begin {While stiff points to expand:} 

q=min(L) {Remove minimum cost pixel q from active list.} 

e(q)=TRUE; {Mark q as e x pa nded( i.e., processed).} 

for each reN(q) such that not e(r) do begin 
gtmp=g(q)+l(q,r); {Compute total cost to neighbor.} 

if reL and gtmp < g(r) then {Remove higher cost neighbor's} 

r=L; { from list} 

if !(reL) then begin {If neighbor not on list,} 

g(r)=gtmp; { assign neighbor's total cost,} 

p(r)=q; { set (or reset) back pointer,} 

L=r; { and place on (or return to)} 

end { active list.} 

end 
end 
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[Name of Document] ABSTRACT „„„ 

dR«NPCT/PTO 12 OCT 2D0I 

[Summary] 
[Task] 

To provide a curve generating apparatus and a curve generating method in 
which in a process of generating intermediate shape of two curves, even in the case 
where contour shape does not linearly move, it is possible to precisely pursue 
correspondence points on the contour. Thus, contour shape with higher precision can 
be prepared. 
[Means for Solution] 

At step S 1 0 1 , user gives, as input, two curves at the time of start and at the time 
of end and reference correspondence points with respect to the respective curves. It 
is required that the numbers of reference correspondence points are the same with 
respect to the two curves. Then, at step S102, pursuit operation of reference 
correspondence point is carried out with respect to picture images successive in the 
time axis direction. As a result, positions of reference correspondence points at 
respective intermediate stages can be detenriined. Then, at step SI 03, interpolation 
of shape is carried out. Interpolation calculation of shape is employed to carry out 
interpolation calculation of coordinates of point trains on two curves to thereby 
determine coordinates of points of intermediate shapes. Then, at step SI 04, 
interpolated shape determined at the step S 103 is deformed in correspondence with 
reference correspondence point detenriined by the pursuit operation at the step SI 02. 
[Selected Drawing] Fig. 3 
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DESCRIPTION 



Curve Generating Apparatus and Curve Generating Method, and Program 
Recording Medium 

Technical Field 

This invention relates to a curve generating apparatus and a curve generating 
method used in CAD (Computer Aided Design), CG (Computer Graphics), etc. which 
edit curve indicating contour shape of object included within picture image on the 
computer, and a program recording medium adapted so that program relating to the 
curve generating method is recorded. 

Background Art 

Hitherto, there are many instances where contour extraction processing for 
extracting contour shape of object included in plural picture image data constituting 
moving picture image are used in the field of picture image processing, e.g., CAD, CG 
technologies, etc. Among these picture image processing, when picture image 
synthesis processing is carried out, there is the processing which generates key signals 
from contour shape of the object, and it is important to generate precise or accurate 
key signals, i.e., precise contour shape. In this case, the key signal is information for 
cutting off the area of the foreground object to be synthesized and is also called mask. 
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The contour extraction processing is required to have interactive operability as in the 
case where operator gives in detail information such as contour position or direction, 
etc. with respect to object. As such contour extraction processing, there are processing 
to designate plural points on the contour of object to prepare shapes between 
respective points, processing to operate control points of parametric curve indicating 
counter shape, and processing to directly input contour shape by mouse, etc. As 
examples of these contour extraction processing, there are known "Intelligent Scissors 
for Image Composition" (Eric N. Mortensen and William A. Barrett, Computer 
Graphics Proceedings, Annual Conference Series, 1995, ACM SIGGRAPH, pp. 191- 
198), "Image contour detection method" (Japanese Patent Application Laid Open No. 
152481/1992 publication), and "Cutting mask preparation method and cutting mask 
preparation apparatus" (Japanese Patent Application Laid Open No. 254854/1992 
publication). 

Moreover, in the contour extraction processing, it is required to extract precise 
contour shape with respect to all picture images constituting moving picture. For 
example, in cinemas or television picture images, several hundreds of key signals are 
required for generating images of several seconds, and the quantity of processing is 
vast. For this reason, the contour extraction processing used in the field of cinema, 
etc. is desired (required) that more precise contour shape can be prepared by lesser 
number of operations. This results from the fact that, in the conventional contour 
extraction processing, processing for preparing contour shapes with respect to 
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respective frames constituting moving picture is the processing which requires 
extremely much time and labor. 

In view of the above, as a method of preparing contour shape by more simple 
procedure, there was proposed a method in which, on the premise that, in moving 
picture images having continuous property, contour shape at the start frame changes 
with passage of time so that it changes to contour shape at the end frame, intermediate 
shape of contour at the intermediate frame is prepared by interpolation processing 
from contour shape at the start frame and contour shape at the end frame. 

In accordance with such an interpolation processing, as shown in FIG. 1, plural 
points constituting contour shape are caused to have one-to-one correspondence 
relationship at contour shape of the start frame and contour shape of the end frame. 
Namely, when it is assumed that curve indicating contour shape at the start frame (time 
t=0) is curve C 0 , curve indicating contour shape at the end frame (time t=l) is curve 
C l5 and curve changes in point of time, it can be considered that curve C 0 at time t=0 
changes to curve Q at time t=l. In view of the above, when time t at which 
interpolation of two curves is assumed to be T, and coordinate at the point on the curve 
C 0 is put by A and coordinate at the point on the curve Q corresponding to the 
coordinate A is put by B, coordinate C at curve C x which takes required intermediate 
form can be represented as follows. 

C= T • A + (1-T) • B 
In the conventional interpolation processing, contour curve C x at the intermediate 



4 

frame (t=T) was prepared from train of points detenriined by such interpolating 
calculation. 

Further, it is assumed that contour shapes with respect to respective frames are 
constituted by plural third-order Bezier curve. The third-order Bezier curve is defined 
by 

Q(t)=((l-t) 3 )M+3t((l-t) 2 )N+3(t 2 )(l-t)O+(t 3 )P(0 1 1) (0<t< 1) 
as indicated by the article "Computer Graphics PRINCIPLE AND PRACTICE 
SECOND EDITION in C" (Foley, van Dam, Feiner, Hughes, ADDISON WESLEY, 
1996, ISBN 0-201-84840-6). In this formula, M, N, O, P are coordinates indicating 
points on the two-dimensional plane wherein M and P are end points of line segment, 
and N, O are control points. An example of line segment of the third order Bezier 
curve represented by the above-mentioned formula is shown in FIG. 2. In FIG. 2, 
curve indicated by solid line is the third-order Bezier curve, wherein M, P indicated 
by points of black circle are end points, and points N, O indicated by white circle are 
control points. 

In accordance with the formula which defines the above-described Bezier curve, 
locus of the third-order Bezier curve, i.e., shape is represented by coordinates of end 
points and control points indicated by points M, N, O, P and change of time t in FIG. 
2. 

When constituting unit of the third-order Bezier curve shown in FIG. 2 is 
assumed to be segment, it can be said that single curve indicating contour shape is 
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ordinarily composed of plural segments as shown in FIG. 3. In this case, end points 
of adjacent segments are shared so that one curve is constituted as the entirety. 

In the above-described conventional contour extraction processing, in the case 
where intermediate shape is determined from contour shapes of frames before and 
after in point of time, interpolation was carried out in the state where control points R 
and end points S constituting segments in FIG. 3 are caused to have one-to-one 
correspondence relationship. 

However, at frames before and after in point of time, there are instances where 
contour shape does not linearly move. 

In the above-mentioned method, in the case where contour shape move non- 
linearly, shape different from contour shape was prepared. 

In order to avoid such problems, it is conceivable to carry out work to divide 
interval in point of time between the time of start and the end time of end into 
narrower ranges to carry out interpolation processing of shape for a second time, or 
to manually modify the shape, etc. For this reason, there took place the problem that 
labour of user is required in preparing shape, and there takes place fine unevenness of 
shape because shape discontinuously changing is prepared in the time axis direction 
owing to manual modification. 

Disclosure of the Invention 

This invention has been made in view of actual circumstances as described 



6 

above, and its object is to provide a curve generating apparatus and a curve generating 
method in which, in the processing for preparing intermediate shape of two curves, 
even in the case where contour shape does not linearly move, it is possible to precisely 
pursue correspondence points on the contour, thus to prepare contour shapes with 
higher precision. 

For this reason, the curve generating apparatus according to this invention is 
directed to a curve generating apparatus adapted for generating, from curve to be 
generated of first frame, curve at second frame, the apparatus comprising: 
correspondence point detecting means for determining, by using curve to be generated 
of the first frame, correspondence points at the second frame corresponding to 
reference correspondence points set on the curve to be generated of the first frame; 
and curve generating means for generating curve of the second frame passing through 
correspondence points detected by the correspondence point detecting means.. 

Moreover, the curve generating method according to this invention is directed 
to a curve generating method for generating, from curve to be generated of the first 
frame, curve at second frame, the method comprising: a correspondence point 
detection step of determining, by using the curve to be generated of the first frame, 
correspondence points at the second frame corresponding to reference correspondence 
points set on the curve to be generated of the first frame; and a curve generation step 
of generating curve of the second frame passing through the correspondence points 
detected by the correspondence detection step. 
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In addition, the program recording medium according to this invention is 
directed to a program providing medium for providing program relating to curve 
generation processing for generating, from curve to be generated of first frame, curve 
at second frame, wherein there is recorded program comprising: a correspondence 
point detecting step of determining, by using the curve to be generated of the first 
frame, correspondence points at the second frame corresponding to reference 
correspondence points set on the curve to be generated of the first frame; and a curve 
generation step of generating curve of the second frame passing through the 
correspondence points detected by the correspondence point detection step. 

Brief Description of the Drawings 

FIG. 1 is a view for explaining a method of preparing shape of intermediate 
frame by interpolation processing from start frame and end frame. 

FIG. 2 is a view showing an example of line segment of third-order Beizer 

curve. 

FIG. 3 is a view showing single curve indicating contour shape as plural 
segments. 

FIG. 4 is a block diagram showing configuration of a curve generating apparatus 
to which this invention is applied. 

FIG. 5 is a view for explaining moving picture consisting of plural frames to be 
processed at the above-mentioned curve generating apparatus. 
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FIG. 6 is a flowchart showing processing procedure in software program 
relating to a curve generating method of this invention. 

FIG. 7 is a view for explaining a method of carrying out pursuit by using 
absolute value sum of differences of picture images as pursuing method for picture 
image. 

FIG. 8 is a view for explaining first contour curve A and second contour curve 
B used when intermediate curve is generated by the curve generating apparatus. 

FIG. 9 is a view showing intermediate curve generated by the curve generating 
apparatus from first contour curve A and second contour curve B. 

FIG. 1 0 is a flowchart showing processing when intermediate curve is generated 
from the first contour curve A and the second contour curve B. 

FIG. 1 1 is a view for explaining that sampling points are set to determine 
sampling interval. 

FIG. 1 2 is a flowchart showing processing when sampling intervals with respect 
to first contour curve A and second contour curve B are detennined. 

FIG. 13 is a view showing first contour curve A and second contour curve B 
from which sampling interval is obtained. 

FIG. 14 is a flowchart showing processing when resampling processing is 
carried out. 

FIG. 15 is a flowchart showing processing when intermediate curve indicating 
intermediate shape is generated. 



9 

FIG. 16 is a flowchart for explaining processing procedure for deforming 
interpolated shape in correspondence with reference correspondence points. 

FIG. 17 is a view showing a more practical example of processing up to step 

542 of the flowchart shown in FIG. 16. 

FIG. 18 is a view showing a more practical example of processing at the step 

543 and steps subsequent thereto of the flowchart shown in FIG. 16. 

FIG. 19 is a flowchart for explaining operation of another more practical 
example of this invention. 

FIG. 20 is a block diagram showing an example of the configuration of the main 
part of curve generating means for generating curve by the more practical example of 
path search method. 

FIG. 21 is a flowchart showing basic processing procedure of the path search 
method. 

FIG.22 is a view for explaining processing for carrying out path search on the 
basis of the path search method. 

FIG. 23 is a view for explaining internal configuration of path search processing 
optimization means. 

FIG. 24 is a flowchart for explaining processing procedure in the path search 
processing optimization means. 

FIG. 25 is a view for explaining processing relating to passing cost calculation 
parameter of the path search processing optimization means. 
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FIG. 26 is a flowchart for explaining processing procedure in the path search 
means. 

FIG. 27 is a view for explaining, in detail, processing relating to passing cost 
calculation parameter utilizing selective gradient detection. 

FIG. 28 is a flowchart for explaining processing procedure in passing cost 
optimization means. 

FIG. 29A is a view showing one example of passing cost map of path search and 
FIG. 29B is a view showing process where rmnimum cost path calculation means 
determines path when passing cost map is obtained. 

FIG. 30 is a view showing one example of the shortest path search algorithm 
using dynamic programming. 

FIG. 3 1 is a view for explaining in detail processing relating to calculation range 
optimization of path search processing optimization means. 

FIG. 32 is a flowchart for explaining processing procedure in calculation range 
optimization means. 

FIG. 33 is a block diagram showing an example of the configuration of point 
train generating means. 

FIG. 34 is a flowchart for explaining processing procedure in the point train 
generating means. 

FIG. 35 is a block diagram showing the configuration of a more practical 
example of optimum color projection axis calculating unit for determining color 
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projection axis. 

FIG. 36 is a view for explaining the operation principle of a more practical 
example of the optimum color projection axis calculating unit. 

FIGS. 37A and 37B are views for explaining the operation principle of another 
more practical example of the optimum color projection axis calculating unit. 

Best Mode for Carrying Out the Invention 

Preferred embodiments of this invention will now be described in detail with 
reference to the attached drawings. 

A curve generating apparatus 1 to which this inventionb is applied is caused to 
be of configuration as shown in FIG. 4, for example. This curve generating apparatus 
1 is adapted so that a program memory 3, a data memory 4, a frame memory 5, an 
operation input unit 6, and an external storage (memory) unit 7 are connected to an 
arithmetic processing unit 2 through a bus 8. 

In the program memory 3, there is recorded program for executing a curve 
generating method according to this invention. Namely, the program memory 3 
records therein, as software program, the curve generating method comprising a 
correspondence point detection step of determining, by using curve to be generated or 
processed of the first frame, correspondence points at second frame corresponding to 
reference correspondence points set on the curve to be generated of the first frame, 
and a curve generation step of generating curve of the second frame passing through 
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correspondence points detected by the correspondence point detection step. 

The operation input unit 6 is comprised of, e.g., keyboard and/or mouse, etc. 
and serves to generate an operation input signal as the result of the fact that it is 
operated by user. This operation input unit 6 outputs the operation input signal to the 
arithmetic processing unit 2. 

The arithmetic processing unit 2 generates control signals with respect to the 
above-described respective units in accordance with operation input signals from the 
operation input unit 6. Further, the arithmetic processing unit 2 outputs control signals 
thus generated to respective units to thereby carry out processing corresponding to the 
operation input signals. Namely, the arithmetic processing unit 2 conducts such a 
control to read out in sequence the software programs from the program memory 3 to 
execute them to generate contour curve indicating contours of curves to be generated 
included in respective frames by using picture data indicating moving picture stored 
in the external recording unit 7. 

Picture image data processed by the software* program from the program 
memory 3 is stored in the external storage unit 7 as described above. The external 
storage unit 7 comprises recording medium, e.g., optical disc, etc. and outputs, in 
accordance with control signal from the arithmetic processing unit 2, picture data 
indicating moving picture recorded on the recording medium to the arithmetic 
processing unit 2, the data memory 4 and the frame memory 5. In addition, the 
external storage (memory) unit 7 records, onto the recording medium, other picture 
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data delivered from the external. 

As shown in FIG. 5, for example, picture image data stored in the external 
storage unit 7 consists of plural frames Frl, Fr2, Fr3, Fr4, Fr5, Fr6 and Fr7 to 
constitute moving picture. In this example, frame Frl is start frame and end frame 
with respect to the frame Frl is frame Fr3. Namely, this curve generating apparatus 
1 executes the program relating to the curve generating method of this invention to 
thereby generate intermediate curve with respect to frame Fr2 which is intermediate 
frame by using first contour curve of data to be generated included in the frame Frl 
and second contour curve of data to be generated included in the frame Fr3. 

In the data memory 4 of FIG. 4, there is stored contour curve indicating contour 
of intermediate frame generated by the above-described arithmetic processing unit 2 
for executing the program, etc. In addition, into this data memory 4, data in the middle 
of arithmetic processing by the arithmetic processing unit 2 is also stored. 

Picture image data corresponding to one frame from the data memory 4 or the 
external storage (memory) unit 7, etc. is inputted to the frame memory 5 in accordance 
with control signal of the arithmetic processing unit 2. Picture image data which has 
been read out by control signal from, e.g., arithmetic processing unit 2 or contour 
curve indicating intermediate frame generated by the program, etc. is inputted to the 
frame memory 5. Further, this frame memory 5 is connected to a display unit 9 for 
displaying picture image and/or processing content, etc. to user, and outputs picture 
image data, etc. inputted in accordance with control signal to the display unit 9 to 
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thereby allow the display unit 9 to display thereon picture image that the above- 
mentioned picture image data indicates. 

The program to which the curve generating method of this invention is applied 
is operated by the processing procedure shown in FIG. 6. 

At step S 1 0 1 , user gives, as input, two curves at the time of start and at the time 
of end and reference correspondence points with respect to respective curves. In this 
case, two curves are respectively referred to as curves A and B. It is required that the 
numbers of reference correspondence points are the same with respect to the curves 
A and B. Then, at step S 1 02, pursuit operation of reference correspondence points is 
carried out with respect to picture images successive in the time axis direction. As a 
result, positions of reference correspondence points at respective intermediate stages 
can be determined. The detail thereof will be described later. 

Then, at step S 103, interpolation of shape is carried out. In order to carry out 
shape interpolation, such an approach is employed to carry out interpolation 
calculation of coordinates of point trains on two curves that user designates at the step 
S101 to thereby determine coordinates of points of intermediate shapes. Although 
there may be employed any transformation (conversion) to provide intermediate shape 
of two shapes, there is employed in this example the technology disclosed in the 
Japanese Patent Application No. 251492/1998 by the applicant of this application. 
The detail thereof will be described later. 

Then, at step SI 04, the interpolated shape determined at the step SI 03 is 
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deformed in correspondence with reference correspondence points determined by 
pursuit operation at the step S102. 

Accordingly, by executing the program, it can be equivalently said that the 
arithmetic processing unit 2 comprises a correspondence point detecting section for 
deteraiming, by using curve to be generated or processed of the first frame, 
correspondence points at intermediate frame corresponding to reference 
correspondence point set on the curve to be generated of frame at the time of start, and 
a curve generating section for generating curve of intermediate frame passing through 
correspondence point detected by the correspondence point detecting section. 

Particularly, the above-mentioned correspondence point detecting section 
carries out picture image pursuit operation at the step SI 02 to detect where reference 
correspondence point on curve to be generated or processed of frame at the time of 
start is positioned at the intermediate frame. 

In addition, the curve generating section generates curve passing through 
correspondence points detected by the correspondence point detecting section through 
the steps SI 03 and SI 04. In more practical sense, when frame at the time of end is 
assumed to be the third frame, interpolated curve is determined by linear interpolation 
from curve to be generated or processed of the first frame and curve to be generated 
or processed of the third frame to deform the interpolated curve thus obtained into 
curve passing through correspondence points detected at the correspondence point 
detecting section. 
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Operations at the respective steps of FIG. 6, i.e., processing that the arithmetic 
processing unit 2 carries out will now be described in detail. 

At step S102, there is carried out an operation to pursue position of reference 
correspondence points in picture images successive on the time axis. As a pursuit 
method of picture image, there may be employed any methods of pursuing points on 
the picture image. Here, explanation will be given with reference to FIG. 7 in 
connection with a method of carrying out pursuit by using absolute value sum of 
differences of picture images, which is known as SSD as an example. When picture 
image around reference correspondence point 10 at pursuit source frame F 0 shown in 
FIG. 7 is assumed as pursuit source image 20, pursuit destination images 40 are 
prepared with respect to all coordinates within the region of search range 30 
designated at frame F l which is pursuing at present shown in FIG. 7 to detennine by 
calculation analogousness between two images, i.e., pursuit source image 20 l5 and 
pursuit destination image 40 v As a method of detenriining analogousness between 
images, there is used a method of determining absolute value sums of differences of 
respective pixel values within image region with respect to two images 20j and 40 2 to 
allow them to be evaluation value to allow coordinates having the lowest evaluation 
value within the region to be pursuit result. 

Explanation will now be described in connection with interpolation calculation 
of shape at step SI 03 will be described below. In FIG. 8, there are shown two curves 
where the numbers of segments are not in correspondence with each other. An 
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example for generating intermediate curve when change from first contour curve A to 
second contour curve B takes place will now be described. In this case, when end 
point, control point and line segment of third-order Bezier curve are assumed to be one 
segment, the first contour curve A consists of five segments Sm and the second 
contour curve B consists of three segments Sm. In addition, when intermediate curve 
in the case where change from first contour curve A to second contour curve B takes 
place as shown in FIG. 9, for example, is generated (indicated as intermediate curve 
group Z c to be generated in FIG. 9), procedure shown in the flowchart of FIG. 10 is 
executed. 

When the intermediate curve is generated, correspondence points indicating 
corresponding positions of the first contour curve A and the second contour curve B 
are designated by operation of keyboard, etc. by user. Thus, at step SI, 
correspondence points are set at the first contour curve A and the second contour 
curve B in accordance with the operation to the effect that the above-described 
correspondence points are designated to develop the first contour curve A and the 
second contour curve B into one straight line as shown in FIG. 11 to determine 
respective entire lengths of the first and second contour curves A and B which are 
closed curve to allow the respective entire lengths of the first and second contour 
curves A and B to be length A and length B, respectively. 

Then, at step S2, sampling intervals of the entire length A of the first contour 
curve A and the entire length B of the second contour curve B are determined. 
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Namely, such an approach is employed to compare magnitudes (sizes) of the length 
A and the length B to divide one longer in length by constant of the sampling interval 
to determine the number of points within the section (step S 1 1 of the flowchart shown 
in FIG. 12). In this case, constant of the sampling interval is sampling interval where 
there can be obtained the number of points necessary for processing for re- 
constructing curve which will be described later at processing for generating such 
intermediate shape. Then, such an approach is employed to divide smaller one of 
magnitudes of length A and length B by the number of sampling points determined at 
the step Sll to obtain sampling interval (step S12 of FIG. 12). For this reason, 
sampling operations are carried out by sampling intervals less than the maximum 
sampling interval Lmax with respect to both curves at all times. 

Reftiming to FIG. 10, at step S3, resampling processing of curve A and curve 
B are carried out at sampling intervals determined at the step S2 to prepare point 
trains. Such state is shown in FIG. 13. FIG. 13 shows the state where resampling 
processing are carried out with respect to the first contour curve A and the second 
contour curve B by using sampling intervals obtained at step Sll. In this case, points 
on segment of one Bezier curve can be represented as a function of time t as described 
below. 

Q(t) = ((l-t) 3 )M+3t(l-t) 2 )N+3(t 2 )(l-t)OH-(t 3 )P (0 < t ^ 1) 
In the above formula, M, N, O, P are coordinates indicating points on the two- 
dimensional plane, wherein M, P are end points of line segment, and N, O are control 
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points. Further, in order to represent, by t, points on the Bezier curve constituted by 
connecting plural segments, definition of the Bezier curve is extended to allow the 
integral part of t to be segment No. and to allow the decimal part oft to be time within 
segment. Namely, in the case of t=3.45, this indicates coordinate on the curve of the 
third segment and at time of 0.45. Thus, points on the Bezier curve constituted by 
plural segments are represented by t. 

Moreover, detailed flowchart of resampling processing carried out at the step 
S3 of FIG. 10 is shown in FIG. 14. When resampling processing is carried out, times 
at correspondence points of the first contour curve A and the second contour curve B 
corresponding to correspondence point O shown in FIG. 13 are set as Ta and Tb. At 
the subsequent step S22, coordinates of points on the first contour curve A and the 
second contour curve B at times Ta, Tb are determined. Respective points determined 
here result in sampling points at the first contour curve A and the second contour curve 
B. At the subsequent step S23, sampling interval detennined at the above-described 
step S2 of FIG. 10 is added to times Ta, Tb. At the subsequent step S24, there is 
judged whether or not times Ta, Tb reach terminating ends so that correspondence 
points return to correspondence points at the time of start of resampling processing. 
Further, when it is judged that correspondence point returns to correspondence point 
at the time of start, the processing is completed. On the other hand, when it is judged 
that correspondence point does not return to the correspondence point, the processing 
returns to step S22 to repeat processing of the above-described steps S22 to S24 to set, 
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for a second time, sampling points at the first contour curve A and the second contour 
curve B to generate point train consisting of sampling points. Namely, in the 
resampling processing of the step S3 of FIG. 10, there are generated point train 
corresponding to the first contour curve A and point trains corresponding to the second 
contour curve B which consist of sampling points finer than those at the above- 
described sampling processing. 

Further, returning to FIG. 10, at step S4, such an approach is employed to carry 
out interpolation calculation of coordinates of point trains on two curves prepared at 
the previously mentioned step S3 to thereby determine point train of intermediate 
shape. In more detail, processing shown in flowchart of FIG. 15 is carried out to 
thereby generate intermediate curve indicating intermediate shape when there takes 
place change from the contour shape that the first contour curve A indicates to contour 
shape that the second contour curve B indicates. In accordance with the flowchart of 
FIG. 15, first, at step S31, point train corresponding to the first contour curve A and 
point train corresponding to the second contour curve B which are determined by 
resampling processing of the step S3 of FIG. 1 0 are inputted from the data memory 4. 
Further, there are designated sampling points at positions corresponding to 
correspondence points on point train corresponding to the first contour curve A and 
point train corresponding to the second contour curve B from the correspondence 
points. At the subsequent step S32, when time at which interpolation of two curves 
is carried out is assumed to be T with respect to point train consisting of sampling 



21 

points, coordinate of point on the first contour curve A is assumed to be a, and 
coordinate of point on the second contour curve B corresponding to the coordinate a 
is assumed to be b, coordinate C of the point at intermediate curve indicating the 
intermediate shape to be determined is expressed by the following formula (2). 

C = T-a + (l-T)-B (2) 
Namely, by applying the above-mentioned formula (2) to point trains corresponding 
to first and second contour curves, coordinate of sampling points constituting 
intermediate curve is determined from sampling points corresponding to the first 
contour curve A and sampling points corresponding to the second contour curve B 
corresponding to sampling points corresponding to the first contour curve A. At the 
subsequent step S33, processing for designating next sampling points of the sampling 
points where the formula (2) is applied at the above-described step S32 is carried out 
with respect to point train corresponding the first contour curve A and point train 
corresponding to the second contour curve B. At the subsequent step S34, the 
arithmetic processing unit 2 judges whether or not correspondence point returns to the 
correspondence point where generation processing of intermediate curve is started at 
the above-described step S31 as the result of the fact that next sampling point is 
designated at the above-described step S33. Further, when it is judged that 
correspondence point returns to the corresponding point, the processing proceeds to 
step S35. On the other hand, when it is judged that correspondence points does not 
return to the correspondence point, the processing which has been explained at the step 
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S32 is carried out with respect to sampling points designated at the step S33. Namely, 
the processing which has been explained at the step S32 is carried out with respect to 
all of sampling points set at the above-described sampling processing of the step S3 
to thereby determine coordinate of points at the intermediate curve in accordance with 
the above-mentioned formula (2). At step S35, by using sampling points determined 
at the above-described step S3 and coordinate C of point at the intermediate curve 
determined at the step S32, the technology shown in the Japanese Patent Application 
Laid Open No. 164436/1998 publication or "An Algorithm for Automatically Fitting 
Digitized Curves (pp. 612-626) in the "Graphics Gems" (Andrew S. Glassner) is 
applied to thereby transform (convert) point train consisting of sampling points into 
Bezier curve. Thus, curve is reconstructed from point train consisting of sampling 
points corresponding to first contour curve A and point train consisting of sampling 
points corresponding to the second contour curve B to generate intermediate curve 
indicating intermediate shape when change from first contour curve A to second 
contour curve B takes place as shown in FIG. 9. 

The curve generating apparatus 1 for carrying out such an intermediate curve 
generation processing is adapted so that when, as shown in FIG. 8, intermediate curve 
when there takes place change from the first contour curve A to the second contour 
curve B is generated, even if the numbers of segments of the first contour curve A and 
the second contour curve B are different from each other, the first and second contour 
curves are caused to undergo sampling to set the same number of sampling points with 
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respect to the first contour curve A and the second contour curve B to re-construct 
curve. Accordingly, it is possible to generate intermediate curve group Zc as shown 
in FIG. 9 with high quality and simply. 

Then, at step SI 04 of FIG. 6, interpolated shape determined at step SI 03 is 
deformed in correspondence with reference correspondence points deteraiined at the 
step SI 02. Explanation will be given with reference to FIGS. 16, 17 and 18 with 
respect to such an operation. FIG. 16 is the detail of deformation processing of the 
step SI 04, FIG. 17 is a view used for explanation from the start START up to the step 
S42 thereof, and FIG. 1 8 is a view used for explanation from the step S43 up to the 
end END. 

First, at step S41 of FIG. 16, start point 115 and end point 1 16 are caused to 
correspond to each other with respect to reference correspondence points 110 
indicated at START of FIG. 17 determined by image pursuit processing of the step 
S102 of FIG. 6 and reference correspondence points 100 indicated at START of FIG. 
17 determined at the intermediate shape preparation processing of the step SI 03. It 
is now assumed that coordinates of start points at the step S103 are A (exl, eyl) and 
B(ex2, ey2) and coordinates of end points at the step S102 are C(sxl, syl) and D(sx2, 
sy2). At the step S42, such a coordinate transformation (conversion) to convert line 
segment AB into line segment CD is determined. Such a transformation can be 
represented by combination of parallel displacement and rotation, and operation of 
scaling. Such a transformation is called affine transformation 120. 
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The transform formular of affine transformation is as follows. 
With respect to the method of transforming point (x, y) into point (x\ y') by 
the affme transformation, the following relational expression is assumed to hold, 
[x'] = sc* [cosO - sin9][x-exl] + [sxl] 
[y'] = sc* [sine - cos9][y-eyl] + [syl] 
In the above formula, sc = L1/L2 
LI = v / ((sx2-sxl) /N 2 + (sy2-syl) A 2) 
L2 = V((ex2-exl) A 2 + (ey2-eyl)^2) 
6=92-91 

91 = a tan 2 ((sy2-syl)/Ll, (sx2-sxl)/Ll) 

92 = a tan 2 ((ey2-eyl)/Ll, (ex2-exl)/L2) 

Then, at the step S43, affine transformation determined at the step S42 is 
calculated with respect to point train constituting intermediate shape determined at the 
step SI 02 of FIG. 6. As a result, as shown in FIG. 18, there is prepared shape which 
holds feature of intermediate shape detennined at the step SI 03 and passing through 
end point determined at the step 102. Then, at step S44, by using the method of 
transforming (converting) point train determined at step S43 into the previously 
described Bezier curve, transformation into curve is carried out. By carrying out such 
a processing with respect to all sections of shape, it becomes possible to transform 
shape prepared at the intermediate shape preparation processing of the step S103 into 
shape passing through reference correspondence points prepared at the step SI 02. 
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As described above, in accordance with this invention, in preparing intermediate 
shape of two curves, it becomes possible to prepare intermediate shape passing 
through designated reference correspondence points. In addition, it is possible to set 
reference correspondence point at precise position by image pursuit processing. As 
a result, it has become possible to prepare, with high precision, intermediate shape 
determined at picture image designated on the time axis. Thus, positional shift 
between shape by linear interpolation and actually determined shape which was the 
problem of the prior art can be solved. 

Another more practical example of this invention will now be described. In 
FIG. 19, there is shown the example in which the step SI 03 is omitted among the 
condition of the first embodiment shown in FIG. 6 and the portion of the step S 104 is 
realized by another method as step SI 05. 

Edge information of picture image is used as substitutive means for preparing 
intermediate shape to carry out processing for automatically generating shape passing 
through the reference correspondence point and along the edge. In more practical 
sense, as a method of detecting optimum path passing through the designated point, 
there is used the technology of path search method disclosed in the Japanese Patent 
Application No. 204862/1997 by the applicant of this application. 

Explanation will be first given in connection with curve generating means which 
generates curve by the path search method disclosed in the Japanese Patent 
Application No. 204862/1997. 
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FIG. 20 is a block diagram showing curve generating means. The curve 
generating means is composed of respective processing means composed of two point 
selector means 151, path search processing optimization means 152, path search 
means 153 and curve approximate means 154, and respective data stores (storages) 16, 
17, 23, 18 for point train, picture image, path and curve. 

The two point selector means 151 selects successive two points P l5 P 2 from the 
point train data storage means 16 to output them. The path search processing 
optimization means 152 calculates path search parameter P rl for optimizing the path 
search method from the selected two points P l5 P 2 and picture image data D v from the 
picture data storage means 17 to send it to the path search means 153. 

The path search means 153 determines path search processing method on the 
basis of the path search parameter P rl that the path search processing optimization 
means 152 has calculated to calculate path data connecting the selected two points P l5 
P 2 and passing on the contour within picture image to hold it by the path data storage 
means 23. 

In this case, path data is list of pixel coordinates indicating eight vicinity 
graphics connecting the above-mentioned two points. As indicated by broken lines in 
FIG. 20, the path search means 153 sends control information I CN to the two point 
selector means 151 to select the next two points. Accordingly, path searches are 
carried out with respect to portions between respective points of the point train storage 
means 16 so that all the search data are accumulated (stored) into the path data storage 
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means 23. 

When the two point selector means 151 has selected until the last of the point 
train to fail to output new two points, it sends control information I CN to the curve 
approximate means 1 54 as indicated by broken lines in FIG. 20 to hasten the operation 
of the curve approximate means 154. The curve approximate means 154 carries out 
curve approximation of path stored in the path data storage means 23 to prepare curve 
data indicating contour shape to hold it by the curve data storage means 18. 

Explanation will now be given with reference to FIGS. 2 1 and 20 in connection 
with path search method that the curve generating means executes. 

FIG. 2 1 is a flowchart showing fundamental processing procedure of the path 
search method according to this invention. First, at step SI, the two point selector 
means 151 selects successive two points from the point train data storage means 16 to 
output them. 

Then, at step S2 5 optimization of path search parameter is carried out. Path 
search parameter P rl is calculated by the path search processing optimization means 
152 on the basis of the two points P l5 P 2 selected by the two point selector means 151 
and picture data Dv. 

Then, at step S3, path search is carried out. The path search means 153 
determines the path search method on the basis of path search parameter obtained at 
the second step S2 to extract contour connecting the above-mentioned two points in 
the form of 8 vicinity path. 
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Further, at step S4, there is carried out judgment as to whether or not path 
searches between all relaying (repeating) points have been completed. As a result, if 
such path searches have not yet been completed, the processing returns to step SI to 
select two points in which one point is shifted from the two points selected at last time 
to repeat steps S2, S3. On the other hand, when it is judged at step S4 that path 
searches between all relaying points have been completed, the processing proceeds to 
step S5. 

At the step S5, contour curve which approximates 8 vicinity path is generated 
to complete the processing. The curve approximate means 154 carries out curve 
approximation of path data stored in the path data storage means 23 to prepare curve 
data indicating contour shape. 

In this case, the curve approximate means 154 can be realized by the technology 
in which the applicant of this application has disclosed "Curve Generating Apparatus 
and Curve Generating Method" in the Japanese Patent Application No. 303687/1996 
publication. This technology is a method of obtaining curve which approximates 
coordinate list and is one of the so-called curve approximation methods. Namely, this 
method is method of approximating order of input data while dividing it within the 
sufficiently approximatable range so that this method is robust to noise and has good 
efficiency as compared to the conventional curve approximations. 

However, the above-mentioned curve approximation means 154 may be 
methods of curve-approximating a series of coordinate lists, and may be realized by 
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a method of generating B-spline curve generally used, etc. 

It is to be noted that general curve approximation/interpolation methods are 
described in detail in articles such as "Numerical Calculation Handbook" (Yutaka 
Ohno, Kazuo Isoda, Ohm Company, 1990), "Shape Processing Engineering [I] [II] by 
Computer Display" (Fujio Yamaguchi, Nikkan Kogyo Shinbun Sha, 1982, "Computer 
Graphics Principles and Practice" (Foly, vanDam, Feiner, and Hughes, Addison 
Wesley Company, 1990), etc. 

The path search processing optimization means 152 and the path search means 
153 will now be described in more detail. 

FIG. 22 is a block diagram in which the portion for carrying out path search is 
extracted of the curve generating means. The feature of the curve generating means 
for carrying out this path search resides in that path search processing optimization 
means 152 is provided to thereby carry out path search processing optimized with 
respect to inputted two points. In this example, the path search processing 
optimization means 152 is adapted to output two kinds of path search parameters P r2 
and P r3 which will be described later. 

FIG. 23 shows the internal configuration of the path search processing 
optimization means 152. The path search processing optimization means 152 is 
composed of passing cost calculation range detennining means 156 and passing cost 
calculation optimization means 157. The passing cost calculation range determining 
means 156 inputs selected two points P 1? P 2 to output calculation range parameter P r3 . 
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Further, the passing cost calculation optimization means 157 outputs passing cost 
calculation parameter P r2 from the selected two points P l5 P 2 and picture data D v . In 
addition, these two kinds of path search parameters P r2 and P r3 are inputted to the path 
search means 1 53 and are used for deterrmning the path search method. 

FIG. 24 is a flowchart showing processing procedure at the path search 
processing optimization means 152. 

First, at step SI 1, optimization of passing cost calculation paraneter is carried 
out. Namely, the passing cost calculation optimization means 157 calculates passing 
cost calculation parameter P r2 from the selected two points (point Pj and point P 2 ) and 
picture image data D v . Then, at step S 12, optimization of the passing cost calculation 
range parameter is carried out to complete the processing. Namely, the passing cost 
calculation range deterrnining means 156 inputs selected two points P l5 P 2 to calculate 
passing cost calculation range parameter P r3 . 

FIG. 25 is a block diagram showing, in more detail, the portion for carrying out 
processing relating to passing cost calculation parameter P r2 in FIG. 23. 

The passing cost calculation optimization means 157 is caused to be of 
configuration comprising gradient parameter determining means 158. 

Moreover, the path search means 153 is caused to be of configuration 
comprising passing cost calculation means 159, nrniimum cost path calculating means 
161, and passing cost map storage means 1 60 for storing data called passage cost map 
which will be described later. 
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The gradient parameter detenriining means 158 inputs picture data D v and two 
points P 1? P 2 to calculate gradient parameter P r4 (a sort of passing cost calculation 
parameters) optimum for extracting contour between the two points to output it to the 
passing cost calculating means 159 of the path search means 153. 

The passing cost calculating means 159 inputs picture data D v , the point P l5 the 
gradient parameter P r4 , and calculation range parameter P r3 that the passing cost 
calculation range determining means 156 shown in FIG. 23 outputs to calculate 
passing cost map to output it to the passing cost map storage means 160. 

The minimum cost path calculating means 161 inputs passing cost map from the 
passing cost map storage means and the point P 2 to calculate path data D p connecting 
the points Pj and P 2 so as to trace the contour therebetween. 

FIG. 26 is a flowchart showing processing procedure at the path search means 

153. 

First, at step S2 1 , the passing cost calculating means 159 calculates passing cost 
map. Then, at step S22, the minimum cost path calculating means 161 calculates the 
minimuin cost path. The detail of this operation will be described later. 

FIG. 27 shows, in more detail, the configuration of the portion for carrying out 
processing relating to the above-described passing cost calculation parameter. 

The passing cost calculation parameter P r2 is two kinds of parameters necessary 
in calculating gradient of picture image. The first parameter is normal vector Y 1 
indicating direction perpendicular to advancing direction of contour, and the second 
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parameter is pixel value color change vector V 0 indicating change of pixel value when 
observed in a direction traversing the contour. 

Moreover, the path search means 153 carries out a processing to extract, by 
utilizing the fact that object contour has high gradient strength, such a path on the 
picture image to pass as close as possible the portion having higher gradient strength. 

In order to carry out processing as described above, the pass cost calculating 
means 159 is composed of gradient calculating means 164, passing cost map 
calculating means 165 and gradient data storage means 21. 

The gradient calculating means 164 calculates gradient of picture image. The 
passing cost map calculating means 165 calculates passing cost map such that when 
path is passed through the portion where gradient strength is higher, cost is reduced 
to more degree on the basis of the gradient thus obtained. Position of the point P 1 is 
inputted to the passing cost map calculating means 165 and starting point of the 
passing cost map is set at the point P v 

In this example, the gradient calculating means 164 selectively detects only 
gradient component of necessary contour by using the technology in which the 
applicent of this application has disclosed in "Edge Detection Method and Edge 
Detection Apparatus" in the Japanese Patent Application No. 134420/1995 
publication. By this gradient calculating means 164, gradient is calculated so as to 
selectively extract gradient of contour extending in a manner connecting inputted two 
points. 
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Explanation will now be given in detail in connection with calculation process 
of the above-described two kinds of path search parameters. 

First, calculation of passing cost calculation parameter P r2 and the process 
where such parameter is utilized will be described. FIG. 28 is a flowchart showing 
processing procedure at the gradient parameter deteniuning means 158. At step S31, 
normal vector calculating means 162 calculates normal vector Vj perpendicular to the 
direction connecting inputted points Pi and P 2 to output it. At step S32, pixel value 
change vector calculating means 163 calculates change of pixel value of picture image 
when observed in a direction perpendicular to the direction connecting two points, e.g. , 
color change direction in the neighbouring region of the points P x and P 2 on the picture 
image to output it as color change vector V 0 . 

In this example, the passing cost map is information in which,in each pixel, 
direction admitted to that pixel position at irrmimum passing cost is recorded. 

FIG. 29 shows an example of simple passing cost map. 

The passing cost map is prepared with respect to one range within picture image 
designated by the passing cost calculation range parameter P r3 . In this example, there 
is shown an example where such map is prepared in a certain small quadrilateral 
region. 

In FIG. 29A, directions admitted to that pixel position at the minimum passing 
cost in the respective pixels are indicated by arrow. This direction is calculated so that 
only one direction is determined in each pixel. Value labeled at each pixel is 
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accumulated value of passing cost when admitted in accordance with the minimum 
passing cost direction. This accumulated value is determined by allowing point 
POINT 1 serving as starting point of path search as accumulated value 0. 

FIG. 29B shows process where minimum path calcualting means 161 
determines path when passing cost map is obtained as shown in FIG. 29A. 

The minimum cost calculating means 161 inputs passing cost map and the point 
POINT 2 to trace, in an opposite direction, minimum passing cost direction of the 
passing cost map from the point POINT 2 to thereby determine path of the minimum 
cost ranging from the point POINT 1 to the point POINT 2. For example, in the case 
where the point POINT 2 is located at position shown in FIG. 29 (accumulated cost 
is 22), the arrows are traced from the point POINT 2 (written in a direction opposite 
to that in FIG. 29A) to determine path until the point POINT 1 as indicated by slanting 
lines. 

Explanation will now be given in connection with means for calculating passing 
cost map as described above at a high speed. Calculation of passing cost map is 
generally known as a sort of optimization problem, and a method of calculating such 
a cost map by using dynamic programming, etc. is well known. Also in the previously 
described article "Intelligent Scissors for Image Composition", a method using 
dynamic prograrmning is described. 

FIG. 30 shows an example of description relating to shortest path search 
algorithm using dynamic prograrmning. It is to be noted that, in this description, signs, 
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etc.. are partially changed from the program described in the above-mentioned article 
"Intelligent Scissors for Image Composition". 

Calculation of calculation range parameter and process where such parameter 
is utilized will now be described. 

FIG. 3 1 is a block diagram showing, in more detail, the portion for carrying out 
processing relating to calculation range parameter in FIG. 23. 

The passing cost calculation range detennining means 156 is caused to be of 
configuration comprising distance calculating means 167, calculation range width 
calculating means 168, calculation range length calculating means 169, and calculation 
range calculating means 170. 

The distance calculating means 167 inputs points P l5 P 2 to calculate distance L 
between the two points and direction G connecting the two points. The calculation 
range length calculating means 169 inputs the distance L between the two points 
obtained to calculate calculation range length SL. The calculation range width 
calculating means 1 68 inputs distance L between the two points obtained to calculate 
calculation range width SW. The calculation range calculating means 170 calculates 
calculation range parameter P r3 to detennine shape of the calculation range on the basis 
of the above-mentioned calculation range length SL, calculation range width SW and 
direction G connecting two points that the distance calculating means 167 has 
outputted to input it to the passing cost calculation means 159 of the path search 
means 153. The passing cost calculation means 159 calculates passing cost map 
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within the range indicated by the calculation range parameter P r3 . 

FIG. 32 is a flowchart showing processing procedure at the passing cost 
calculation range detennining means 156 of FIG. 31. 

First, at step S41, the distance calculating means 167 inputs points Pj and P 2 to 
calculate distance L between these two points and direction G connecting two points. 

Then, at step S42, the distance L between the two points is inputted. Thus, 
calculation range width SW is calculated. In this case, the calculation range width 
calculating means 168 has therein rule of magnitude (size) of calculation range width 
S W with respect to magnitude (size) of distance L between two points in the form such 
as calculation formula or reference table, etc. to determine calculation range width SW 
in accordance with that rule. For example, the problem in which even if operator 
attempts to set relaying (repeating) points in detail at short intervals in order to obtain 
a desired contour, searched path is meandered between relaying points so that shape 
differs from desired shape of contour can be solved by employing a rule such that the 
rule for detenmning calculation range width SW changes to a rule such that according 
as distance between two points becomes smaller, width becomes smaller. 

At this time, calculation range width SW between relaying (repeating) points set 
in detail at short intervals is set so that it becomes very narrow, and path search is 
carried out so that relaying (repeating) points are connected by substantially straight 
line. Thus, contour shape that operator desires is obtained. 

Then, at step S43, the calculation range length calculating means 169 calculates 
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calculation range length SL from distance L between two points. In this case, the 
calculation range length SL is detennined so that it is larger than the distance L 
between two points by margin length determined in advance. 

Finally, at step S44, the calculation range calculating means 170 calculates 
calculation range parameter P r3 from the calculation range width S W, the calculation 
range length SL and direction G connecting two points. In this case, the calculation 
range is such a rectangle to surround inputted two points P l5 P 2 and inclined in the 
direction connecting two points, and the calculation range parameter P r3 indicates 
positions of four corners of that rectangle. 

The point train generating means 12 will now be described. 

The point train generating means 12 inputs curve C to output point train 
reproducing that shape and reconstructed curve reconstructed by that point train. The 
point train and the reconstructed curve which have been generated are used for curve 
editing work which will be carried out later by the configuration of the first 
embodiment. 

FIG. 33 is a block diagram showing an example of the configuration of the point 
train generating means 12. 

This point train generating means 12 is caused to be of configuration 
comprising initial point train generating means 171, curve reconstructing means 172, 
difference detecting means 173, point train editing means 14, picture image data 
storage means 17, point train data storage means 16, and reconstructed curve data 
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storage means 1 9. 

The initial point train generating means 171 inputs input curve C and picture 
data D v to generate initial point train. The point train is stored in the point train data 
storage means 1 6. The curve reconstructing means 172 inputs point train and picture 
image data D v to generate such a curve to pass the point train and to pass on the 
contour of the picture image. 

The difference detecting means 173 inputs input curve C and re-constructed 
curve C to output difference between both shapes as difference dif. The point train 
editing means 14 operates point train data in the point train data storage means 16 on 
the basis of difference dif that the difference detecting means 173 has outputted. 

FIG. 34 is a flowchart showing processing procedure in the point train 
generating means 12. 

First, at step S51, the initial point train generating means 171 carries out 
extraction of feature points of input curve C to store the extracted feature points into 
the point train data storage means 16 as initial point train. 

For the above-mentioned feature point extraction on the curve, there may be 
used feature point extraction method shown in "Motion Vector Estimation Method" 
described in the Japanese Patent Application Laid Open No. 335270/1 996 publication. 
In accordance with this method, such an approach is employed to calculate very small 
section bending angles at respective points from the initial end to the terminating end 
of curve to allow its sum total to be accumulated bending angle, whereby when the 
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accumulated bending angle is greater than a threshold value determined in advance, 
the point which takes the maximum very small section bending angle is caused to be 
feature point. Further, similar feature point extraction processing is recursively carried 
out with respect to curve divided by feature points. 

Then, at step S52, the curve re-constructing means 172 calculates re-constructed 
curve by processing similar to the processing in the previously described path search 
apparatus according to this invention from the point train. 

Then, at step S 5 3, the difference detecting means 173 calculates difference dif 
between input curve and re -constructed curve. In accordance with such difference 
detection processing, such an approach is employed to take difference detection points 
at suitable intervals on the re-constructed curve to measure distance up to input curve 
from that difference detection point to allow it to be difference value dif. 

Then, at step S54, the difference detecting means 173 examines whether or not 
there exist difference value dif greater than threshold value set in advance. If there 
does not exist difference value greater than the threshold value, the processing of the 
pointtrain generating means 12 is completed. Thus, data in the point train data storage 
means 16 and re-constructed curve data storage means 19 at that time point serve as 
output. On the other hand, if there exists difference value greater than the threshold 
value, the processing proceeds to step S55. 

At the step S55, the point train editing means 14 edits point train data on the 
basis of difference dif thus obtained. At respective difference detection points in 
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which it is judged that the difference value dif is greater than threshold value, new 
relaying (repeating) point is placed at point on the input curve nearest from that 
difference detection point to store point train data into the point train data storage 
means 16. 

After completion of the step S55, the processing returns to the step S52. Thus, 
at step S54, processing is repeated until difference between input curve C and re- 
constructed curve C becomes smaller than threshold value. 

While, in the above-mentioned processing, the number of points constituting 
initial point train is set to lesser value at first to suitably supplement initial points at the 
shortage portions, there may be employed a method in which the number of points 
constituting initial point train is set to larger value to delete unnecessary points. In that 
case, such an approach may be employed to generate initial point train to such a degree 
that input curve can be sufficiently polygon-approximated by the initial point train 
generating means 171 to select and delete relaying (repeating) points in which 
difference value therearound is sufficiently smaller than threshold value by the 
difference detecting means 173 after re-construction of the curve. 

It is to be noted that while, in the processing procedure shown in FIG. 34, since 
re-construction can be completed by lesser number of points, there results the merit 
that curve editing thereafter is easy, processing procedure which will be described later 
has the merit that because relaying (repeating) point interval for carrying out path 
search can be small, processing time can be shortened. 
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Moreover, point train obtained by the above-mentioned method is transformed 
(converted) into curve by using curve generating means. Namely, by using edge 
information of picture image in preparing intermediate shape passing through portions 
between respective correspondence points by using the above-mentioned method, it 
has become possible to prepare contour having high precision along the shape 
represented in the picture image. 

Further, in the method shown in FIG. 19, such an approach may be employed 
to determine edge of picture image by using another algorithm at step S105 to 
determine intermediate shape along the edge of that picture image. 

Namely, at step SI 02, a set of correspondence points determined by picture 
image pursuit and the picture image are used to determine color projection axis which 
maximizes contrast around the edge to form edge of picture image by using such color 
projection axis to determine intermediate shape along this edge. 

The configuration of optimum color projection axis calculating unit 32 which 
inputs the set of correspondence points as representative points Ql and Q2 to 
determine the color projection axis P jo is shown in FIG. 35. 

The optimum color projection axis calculating unit 32 is composed of a 
representative point vicinity (neighbouring) color projection axis calculating section 
41a and a representative point vicinity (neighbouring) optimum color projection axis 
calculating section 41b for calculating vicinity (neighbouring) optimum color 
projection axes and P j2 within the region in the vicinity of representative points Ql 
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and Q2, and a color projection axis synthesis section 42 for synthesizing color 
projection axis P j0 by using the color projection axis ? }1 and the color projection axis 
P j2 from the above-mentioned two calculating sections 41a, 41b. 

At the representative point vicinity (neighbouring) optimum color projection 
axis calculating sections 41a and 41b, there are determined vicinity (neighbouring) 
optimum color projection axes and P j2 which maximize contrast around the edge 
included within neighboring regions of representative points Ql and Q2. It is now 
assumed that, as shown in FIG. 36, representative points Ql and Q2 are determined 
within the regions color- separated by black B, white W, green G, yellow Y, red R and 
blue B. At this time, the representative point vicinity (neighbouring) color projection 
axis calculating section 41a deteraiines, by calculation, color projection axis P^ in 
which the contrast is maximum of edge included within the neighboring region Al of 
representative point Ql. In addition, the representative point vicinity neighboring 
color projection axis calculating section 41b determines, by calculation, color 
projection axis P j2 in which the contrast is maximum of edge included within the 
vicinity (neighbouring) region A2 of the representative point Q2. 

The color projection axis synthesis section 42 synthesizes vicinity 
(neighbouring) optimum color projection axis P jl5 vicinity (neighbouring) optimum 
color projection axis P j2 respectively independently determined within the vicinity 
(neighbouring) regions of representative points Ql, Q2 to determine one color 
projection axis P jo where the contrast is maximum of edge of updating section. As the 
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synthesis method in this case, e.g., average vector of color projection axes may be axis 
after synthesis. 

In addition, the maximum color projection axis calculating section 32 may 
deteraiine optimum color projection axis by the method shown in FIG. 37. 

As shown in FIG. 37 A, it is assumed that anchor point ApO and adjacent anchor 
point Apl are placed on edge e of picture image. At this time, since there is the 
possibility that the edge e may deviate to much degree from the line segment 
connecting ApO and Apl, there is carried out search of region where color change is 
the greatest within the range of length H on the line segment in a direction 
perpendicular to ApOApl with respect to three points placed every equal interval with 
respect to line segment length L on the line segment ApOApl, e.g., interval of L/4. 

When color with respect to a certain point r is assumed to be C(r), it is defined 
that color change vector is dC/dr and its magnitude is |dC/dr|. Thus, r in which this 
value takes the maximum value is determined. 

As the result of search, rO, r 1 , r2 are assumed to be determined as shown in FIG. 
37B. At this time, color change vectors dC/dr are respectively determined at rO, rl, 
r2 to allow its average value to be optimum color projection axis. 

In addition, color change vector may be determined in a manner described 
below. In the case of rO, colors C(sOO), C(s01) are determined with respect to points 
sOO, sOl having rO as center and away from each other by distance width h on the 
straight line perpendicular to the line segment ApOApl . Thus, color change vector is 
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substituted for its difference vector aC=C(s01)-C(s00). In connection with rl, r2, 
color change vectors are respectively similarly determined. 

Ordinary contour closed curve generating unit generates contour closed curve 
on the basis of the optimum color projection axis which has been explained. At the 
step SI 05, intermediate shape based on this contour closed curve is prepared. 

In addition, when algorithm shown in FIG. 19 is deformed and linear 
interpolation at the step S 102 fails to be carried out because, e.g., movement of picture 
image is suddenly carried out, the processing of the step SI 05 may be carried out. 

As described above, in accordance with this invention, positional shift between 
shape by linear interpolation and shape actually determined, which was problem of the 
prior art, can be solved. 

Industrial Applicability 

As previously mentioned, in accordance with this invention, even in the case 
where contour shape does not linearly move, it is possible to precisely pursue 
correspondence points on the contour. Thus, contour shape having higher precision 
can be prepared. 
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CLAIMS 

1. A curve generating apparatus adapted for generating, from curve to be 
generated or processed of first frame, curve at second frame, 

the apparatus comprising: 

correspondence point detecting means for detennining, by using the curve to be 
generated of the first frame, correspondence point at the second frame corresponding 
to reference correspondence point set on the curve to be generated of the first frame; 
and 

curve generating means for generating curve of the second frame passing 
through the correspondence point detected by the correspondence point detecting 
means. 

2. The curve generating apparatus as set forth in claim 1, 

wherein the correspondence point detecting means carries out picture image 
pursuit of reference correspondence point set on the curve to be generated of the first 
frame to determine correspondence point at the second frame. 

3. The curve generating apparatus as set forth in claim 1 , 

wherein the curve generating means is adapted so that when the first frame is 
caused to be frame at the time of start and frame at the time of end is caused to be 
third frame, it determines interpolated curve by linear interpolation from curve to be 
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generated of the first frame and curve of the third frame to deform this interpolated 
curve into curve passing through the correspondence point detected by the 
correspondence point detecting means. 

4. The curve generating apparatus as set forth in claim 1, 

wherein the curve generating means generates, as curve, shape along edge 
of picture image passing through correspondence point detected by the 
correspondence point detecting means. 

5. The curve generating apparatus as set froth in claim 1, 

wherein the curve generating means generates contour curve of object on 
picture image. 

6. A curve generating method for generating, from curve to be generated or 
processed of first 

frame, curve at second frame, 
the method comprising: 

a correspondence point detection step of determining, by using curve to be 
generated of the first frame, correspondence point at the second frame corresponding 
to reference correspondence point set on the curve to be generated of the first frame; 
and 
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a curve generating step of generating curve of the second frame passing through 
the correspondence point detected by the correspondence point detection step. 

7. The curve generating method as set forth in claim 6, 

wherein, at the correspondence point detection step, a procedure is taken to 
carry out picture image pursuit of reference correspondence point set on the curve to 
be generated of the first frame to determine correspondence point at the second frame. 

8. The curve generating method as set forth in claim 6, 

wherein, at the curve generating step, a procedure is taken such that when the 
first frame is assumed to be frame at the time of start and frame at the time of end is 
assumed to be third frame, interpolated curve is determined by linear interpolation 
from the curve to be generated of the first frame and curve of the third frame to 
deform this interpolated curve into curve passing through the correspondence point 
detected at the correspondence detection step. 

9. The curve generating method as set forth in claim 6, 

wherein, at the curve generating step, a procedure is taken to generate, as curve, 
shape along edge of picture image passing through correspondence point detected at 
the correspondence point detection step. 
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10. The curve generating method as set forth in claim 6, 

wherein, at the curve generating step, a procedure is taken to generate contour 
curve of object on picture image. 

11. A program recording medium adapted so that there is recorded program 
relating 

to curve generating processing for generating curve at second frame from curve to be 
generated ir oricessed of first frame, 
the program comprising: 

a correspondence point detection step of detennining, by using curve to be 
generated of the first frame, correspondence point at the second frame corresponding 
to reference correspondence point set on the curve to be generated of the first frame; 
and 

a curve generation step of generating curve of the second frame passing through 
the correspondence point detected by the correspondence detection step. 
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ABSTRACT 

At step S 10 1 , user gives, as input, two curves at the time of start and at the time 
of end and reference correspondence points with respect to the respective curves. 
Then, at step S102, pursuit operation of reference correspondence point is carried out 
with respect to picture images successive in the time axis direction. As a result, 
positions of reference correspondence points at respective intermediate stages can be 
determined. Then, at step SI 03, interpolation of shape is carried out. Then, at step 
S104, interpolated shape determined at the step SI 03 is deformed in correspondence 
with reference correspondence point determined by the pursuit operation at the step 
SI 02. Accordingly, even in the case where contour shape does not linearly move, it 
is possible to precisely pursue correspondence points on the contour. Thus, contour 
shape with higher precision can be prepared. 
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The Live-Wire 2-D dynamic programming (DP) graph search algorithm is as follows: 
Algorithm: Live-Wire 2-D DP graph search. 



Input: 

s {Start(or seed) pixel,} 

l(q,r) {Local cost function for link between pixels q and r.} 
Data Structures: 

L {List of active pixels sorted by total cost (initially empty).} 

N(q) {Neighborhood set of q (contains 8 neighbors of pixel).} 

e(q) {Boolean function indicating if q has been expanded/processed.} 

g(q) {Total cost function from seed point to q.} 

Output: 

p {Pointers from each pixel indicating the minimum cost path.} 



Algorithm: 

g(s)=0; L=s; 

while L!=NULL do begin 

q=min(L) 

e(q)=TRUE; 

for each reN(q) such that not 
gtmp=g(q)+l(q,r); 
if reL and gtmp < g(r) then 
r=L; 

if !(reL)then begin 
g(r)=gtmp; 

p(0=q; 

L=r; 
end 
end 
end 
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{Initialize active list with zero cost seed pixel.} 
{While still points to expand:} 
{Remove minimum cost pixel q from active list.} 
{Mark q as expanded(i.e., processed).} 
e(r) do begin 

{Compute total cost to neighbor.} 
{Remove higher cost neighbor's} 
{ from list} 

{If neighbor not on list,} 
{ assign neighbor's total cost,} 
{ set (or reset) back pointer,} 
{ and place on (or return to)} 
{ active list.} 
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